Sep 17 19:03:56.189246 ip-10-0-134-45 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 17 19:03:56.189259 ip-10-0-134-45 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 17 19:03:56.189266 ip-10-0-134-45 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 17 19:03:56.189528 ip-10-0-134-45 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 17 19:04:06.203502 ip-10-0-134-45 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 17 19:04:06.203542 ip-10-0-134-45 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dab7951255cc4ae88bc2d2329429da2d --
Sep 17 19:05:53.225060 ip-10-0-134-45 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 19:05:56.846266 ip-10-0-134-45 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:05:56.846266 ip-10-0-134-45 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 19:05:56.846266 ip-10-0-134-45 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:05:56.846266 ip-10-0-134-45 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 19:05:56.846266 ip-10-0-134-45 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 19:05:56.846953 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.846349    2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 19:05:56.849109 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849094    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.849109 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849109    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849113    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849117    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849120    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849122    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849125    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849127    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849130    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849133    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849135    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849138    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849142    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849146    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849150    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849153    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849156    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849164    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849167    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849170    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.849174 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849173    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849175    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849178    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849181    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849184    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849187    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849190    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849193    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849196    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849198    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849201    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849203    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849206    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849208    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849211    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849213    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849216    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849218    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849221    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849223    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.849642 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849226    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849228    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849231    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849233    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849236    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849238    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849241    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849243    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849246    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849254    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849256    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849259    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849261    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849264    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849268    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849271    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849273    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849276    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849279    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849282    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.850137 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849286    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849289    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849291    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849294    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849296    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849299    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849301    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849304    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849307    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849310    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849313    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849316    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849318    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849321    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849324    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849326    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849329    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849331    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849334    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849336    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.850638 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849339    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849342    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849344    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849347    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849349    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849352    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849356    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849359    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849362    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849364    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849367    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849369    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849372    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849374    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849377    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849379    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849382    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849384    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.849387    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850052    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.851112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850058    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850061    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850064    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850066    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850070    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850072    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850075    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850078    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850080    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850083    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850086    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850088    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850091    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850095    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850099    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850102    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850104    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850107    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850110    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.851590 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850113    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850115    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850118    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850121    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850123    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850125    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850128    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850130    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850133    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850135    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850138    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850140    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850144    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850148    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850151    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850155    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850157    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850160    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850164    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.852104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850167    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850169    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850172    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850175    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850177    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850180    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850183    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850185    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850187    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850191    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850193    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850196    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850199    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850201    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850203    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850206    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850209    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850211    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850214    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850216    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.852560 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850219    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850221    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850224    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850226    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850229    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850231    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850234    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850236    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850239    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850241    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850244    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850246    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850249    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850252    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850255    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850257    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850260    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850263    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850265    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850268    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.853104 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850270    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850273    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850275    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850278    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850280    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850283    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850286    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850288    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850291    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850294    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850298    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850301    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850304    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850307    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850310    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850313    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850315    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850318    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850321    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850324    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.853595 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850704    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850709    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850712    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850715    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850719    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850721    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850724    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850727    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850733    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850736    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850739    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850741    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850744    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850747    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850749    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850752    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850755    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850757    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850760    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.854077 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850763    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850765    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850768    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850770    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850773    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850775    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850778    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850780    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850782    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850785    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850787    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850791    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850794    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850797    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850799    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850802    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850804    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850807    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850809    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.854537 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850813    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850815    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850820    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850822    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850825    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850827    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850830    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850832    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850835    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850839    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850842    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850845    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850849    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850852    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850854    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850858    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850862    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850865    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850868    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.855013 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850870    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850873    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850875    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850878    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850881    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850883    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850886    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850888    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850890    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850893    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850896    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850898    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850900    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850903    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850906    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850910    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850912    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850915    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850917    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850920    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.855468 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850922    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850925    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850927    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850930    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850933    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850935    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850938    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850941    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850943    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850946    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850948    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850951    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850953    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850956    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850958    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850961    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850963    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850966    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850968    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850971    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.855973 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850973    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.850976    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851059    2586 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851106    2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851116    2586 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851121    2586 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851125    2586 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851131    2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851135    2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851138    2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851142    2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851145    2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851148    2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851151    2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851154    2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851157    2586 flags.go:64] FLAG: --cgroup-root=""
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851160    2586 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851163    2586 flags.go:64] FLAG: --client-ca-file=""
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851166    2586 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851168    2586 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851172    2586 flags.go:64] FLAG: --cluster-domain=""
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851175    2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851178    2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851181    2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 19:05:56.856466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851184    2586 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851188    2586 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851191    2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851194    2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851197    2586 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851200    2586 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851202    2586 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851205    2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851208    2586 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851211    2586 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851215    2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851218    2586 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851220    2586 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851223    2586 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851226    2586 flags.go:64] FLAG: --enable-server="true"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851232    2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851236    2586 flags.go:64] FLAG: --event-burst="100"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851239    2586 flags.go:64] FLAG: --event-qps="50"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851242    2586 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851245    2586 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851248    2586 flags.go:64] FLAG: --eviction-hard=""
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851251    2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851254    2586 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851257    2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851260    2586 flags.go:64] FLAG: --eviction-soft=""
Sep 17 19:05:56.857082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851263    2586 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851266    2586 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851268    2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851271    2586 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851274    2586 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851277    2586 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851280    2586 flags.go:64] FLAG: --feature-gates=""
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851283    2586 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851286    2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851289    2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851292    2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851295    2586 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851298    2586 flags.go:64] FLAG: --help="false"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851300    2586 flags.go:64] FLAG: --hostname-override="ip-10-0-134-45.ec2.internal"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851303    2586 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851306    2586 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851309    2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851312    2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851316    2586 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851318    2586 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851321    2586 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851324    2586 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851329    2586 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 19:05:56.857694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851332    2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851335    2586 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851338    2586 flags.go:64] FLAG: --kube-reserved=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851341    2586 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851344    2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851347    2586 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851349    2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851352    2586 flags.go:64] FLAG: --lock-file=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851355    2586 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851358    2586 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851360    2586 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851365    2586 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851368    2586 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851371    2586 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851374    2586 flags.go:64] FLAG: --logging-format="text"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851376    2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851379    2586 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851382    2586 flags.go:64] FLAG: --manifest-url=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851385    2586 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851388    2586 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851391    2586 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851395    2586 flags.go:64] FLAG: --max-pods="110"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851398    2586 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851401    2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851404    2586 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 19:05:56.858312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851406    2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851409    2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851412    2586 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851415    2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851422    2586 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851424    2586 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851429    2586 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851432    2586 flags.go:64] FLAG: --pod-cidr=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851436    2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851440    2586 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851443    2586 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851446    2586 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851449    2586 flags.go:64] FLAG: --port="10250"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851452    2586 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851454    2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01f28607449f75ca7"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851457    2586 flags.go:64] FLAG: --qos-reserved=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851460    2586 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851463    2586 flags.go:64] FLAG: --register-node="true"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851466    2586 flags.go:64] FLAG: --register-with-taints=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851469    2586 flags.go:64] FLAG: --registry-burst="10"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851472    2586 flags.go:64] FLAG: --registry-qps="5"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851475    2586 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851478    2586 flags.go:64] FLAG: --reserved-memory=""
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851481    2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851484    2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 19:05:56.858940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851487    2586 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851489    2586 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851492    2586 flags.go:64] FLAG: --runonce="false"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851495    2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851498    2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851500    2586 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851503    2586 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851506    2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851508    2586 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851512    2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851515    2586 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851517    2586 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851520    2586 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851524    2586 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851528    2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851531    2586 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851534    2586 flags.go:64] FLAG: --system-cgroups=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851537    2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851542    2586 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851545    2586 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851547    2586 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851551    2586 flags.go:64] FLAG: --tls-min-version=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851554    2586 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851556    2586 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851559    2586 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 19:05:56.859556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851562    2586 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851565    2586 flags.go:64] FLAG: --v="2"
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851568    2586 flags.go:64] FLAG: --version="false"
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851572    2586 flags.go:64] FLAG: --vmodule=""
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851575    2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.851579    2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851704    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851709    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851712    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851715    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851718    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851723    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851726    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851729    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851732    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851735    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851739    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851742    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851745    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851748    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.860198 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851752    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851756    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851759    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851761    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851764    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851767    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851770    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851773    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851775    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851778    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851788    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851791    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851794    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851796    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851799    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851802    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851804    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851807    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851810    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851812    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.860706 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851815    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851817    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851820    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851822    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851825    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851827    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851830    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851833    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851835    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851838    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851841    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851843    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851848    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851850    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851853    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851856    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851858    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851861    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851864    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851867    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.861184 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851869    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851872    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851874    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851877    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851879    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851882    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851884    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851887    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851890    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851892    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851895    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851898    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851900    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851903    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851907    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851911    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851914    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851916    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851919    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851922    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.861687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851924    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851927    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851929    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851931    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851936    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851938    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851940    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851943    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851945    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851948    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851951    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851953    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851956    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851958    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851961    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851963    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851966    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851968    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851970    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851973    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.862179 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851975    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851978    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851980    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851982    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.851985    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.852413    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.859294    2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.859309    2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.859328    2586 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.859336    2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859390    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859394    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859398    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859401    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.862687 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859404    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859407    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859409    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859412    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859415    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859417    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859420    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859423    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859426    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859429    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859432    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859434    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859437    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859440    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859443    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859446    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859448    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859451    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859455    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.863042 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859459    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859462    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859465    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859468    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859470    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859473    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859476    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859480    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859482    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859485    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859487    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859490    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859492    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859495    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859497    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859500    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859502    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859505    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859507    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.863498 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859510    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859514    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859518    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859521    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859524    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859527    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859530    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859533    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859535    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859538    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859541    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859544    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859547    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859549    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859552    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859555    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859558    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859560    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859563    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.864112 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859566    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859569    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859572    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859574    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859577    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859579    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859582    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859584    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859587    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859591    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859595    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859597    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859600    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859602    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859605    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859607    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859610    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859613    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859615    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859635    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.864611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859638    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859640    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859642    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859645    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859647    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859650    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859653    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859655    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859658    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859660    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859663    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859665    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859668    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859671    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859674    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859676    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859679    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.865162 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859681    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.859686    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859789    2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859793    2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859795    2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859798    2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859801    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859804    2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859808    2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859811    2586 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859814    2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859817    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859819    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859822    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 19:05:56.865575 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859825    2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859827    2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859830    2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859832    2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859835    2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859837    2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859840    2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859842    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859844    2586 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859847    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859849    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859852    2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859854    2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859857    2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859860    2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859863    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859865    2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859868    2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859870    2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 19:05:56.865995 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859873    2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859875    2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859877    2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859880    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859882    2586 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859885    2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859888    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859891    2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859894    2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859898    2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859900    2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859903    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859905    2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859908    2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859910    2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859913    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859915    2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859919    2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859922    2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 19:05:56.866444 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859925    2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859928    2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859930    2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859932    2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859935    2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859937    2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859940    2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859943    2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859946    2586 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859948    2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859951    2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859953    2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859956    2586 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859959    2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859961    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859963    2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859966    2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859968    2586 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859971    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859973    2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 19:05:56.866943 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859976    2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859978    2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859980    2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859983    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859986    2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859988    2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859990    2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859993    2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859996    2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.859998    2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860000    2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860003    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860005    2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860008    2586 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860010    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860012    2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860015    2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860017    2586 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860019    2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860022    2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 19:05:56.867438 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860025    2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860028    2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860030    2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860033    2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860035    2586 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860037    2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860040    2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860042    2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:56.860045    2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.860049    2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.860148    2586 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.862341    2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.863176    2586 server.go:1019] "Starting client certificate rotation"
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.863279    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:05:56.868095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.863319    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 19:05:56.868448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.866274    2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:05:56.868448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.867385    2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 19:05:56.872104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.872073    2586 log.go:25] "Validated CRI v1 runtime API"
Sep 17 19:05:56.873779 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.873764    2586 log.go:25] "Validated CRI v1 image API"
Sep 17 19:05:56.874513 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.874499    2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 19:05:56.877311 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.877293    2586 fs.go:135] Filesystem UUIDs: map[3fa0f697-0e34-432f-8661-14451e22aa27:/dev/nvme0n1p3 5eae3655-b322-42ed-9ec7-a785af010a97:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Sep 17 19:05:56.877353 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.877312    2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 19:05:56.882403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.882304    2586 manager.go:216] Machine: {Timestamp:2026-09-17 19:05:56.881833571 +0000 UTC m=+2.047574853 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3088184 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f2cbda86252f20cd64addb911dd78 SystemUUID:ec2f2cbd-a862-52f2-0cd6-4addb911dd78 BootID:dab79512-55cc-4ae8-8bc2-d2329429da2d Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c0:07:ec:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c0:07:ec:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0e:83:e8:c0:74:3c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 19:05:56.882486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.882405    2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 19:05:56.882486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.882473    2586 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 19:05:56.884038 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.884012    2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 19:05:56.884181 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.884040    2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-45.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 19:05:56.885044 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.885033    2586 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 19:05:56.885076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.885048    2586 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 19:05:56.885076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.885060    2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:05:56.885076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.885069    2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 19:05:56.885156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.885135    2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 19:05:56.887270 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887258    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:05:56.887516 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887507    2586 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 19:05:56.887552 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887542    2586 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 19:05:56.887582 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887552    2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 19:05:56.887582 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887569    2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 19:05:56.887582 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887577    2586 kubelet.go:398] "Adding apiserver pod source"
Sep 17 19:05:56.887699 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887586    2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 19:05:56.887922 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887908    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:05:56.887965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.887933    2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 19:05:56.890167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.890148    2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 19:05:56.892771 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.892746    2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 19:05:56.892855 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.892792    2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 19:05:56.893819 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893799    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893835    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893844    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893853    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893865    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893873    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 19:05:56.893897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893892    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893901    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893911    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893920    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893936    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893945    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893972    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 19:05:56.894189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.893979    2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 19:05:56.896459 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.896434    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:05:56.898841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.898824    2586 server.go:1268] "Started kubelet"
Sep 17 19:05:56.898942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.898910    2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 19:05:56.898942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.898908    2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 19:05:56.899092 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.898966    2586 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 19:05:56.899212 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.899190    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 19:05:56.899266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.899231    2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-45.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 19:05:56.899403 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.899380    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 19:05:56.899781 ip-10-0-134-45 systemd[1]: Started Kubernetes Kubelet.
Sep 17 19:05:56.901372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.900310    2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 19:05:56.901372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.900448    2586 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 19:05:56.903773 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.903752    2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 19:05:56.905124 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905111    2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 19:05:56.905181 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905124    2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 19:05:56.905228 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.905179    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:56.905552 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905533    2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 19:05:56.905663 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905553    2586 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 19:05:56.905663 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905567    2586 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 19:05:56.905772 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905729    2586 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 19:05:56.905772 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905739    2586 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 19:05:56.906005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.905992    2586 factory.go:55] Registering systemd factory
Sep 17 19:05:56.906055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906006    2586 factory.go:223] Registration of the systemd container factory successfully
Sep 17 19:05:56.906319 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906305    2586 factory.go:153] Registering CRI-O factory
Sep 17 19:05:56.906354 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906323    2586 factory.go:223] Registration of the crio container factory successfully
Sep 17 19:05:56.906394 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906384    2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 19:05:56.906437 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906405    2586 factory.go:103] Registering Raw factory
Sep 17 19:05:56.906437 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906419    2586 manager.go:1196] Started watching for new ooms in manager
Sep 17 19:05:56.906764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.906754    2586 manager.go:349] Starting recovery of all containers
Sep 17 19:05:56.907468 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.907413    2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 19:05:56.907468 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.907492    2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-45.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 19:05:56.916569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.916420    2586 manager.go:354] Recovery completed
Sep 17 19:05:56.920554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.920543    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:56.923131 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.923108    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:56.923199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.923146    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:56.923199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.923160    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:56.927597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.927576    2586 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 19:05:56.927597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.927596    2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 19:05:56.927737 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.927615    2586 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 19:05:56.927737 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.927639    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:56.928559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928538    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:56.928697 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928568    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:56.928697 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928588    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:56.928888 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928872    2586 policy_none.go:49] "None policy: Start"
Sep 17 19:05:56.928971 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928894    2586 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 19:05:56.928971 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.928908    2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 19:05:56.930122 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.930107    2586 policy_none.go:47] "Start"
Sep 17 19:05:56.938847 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.937745    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-45.ec2.internal.18d63129a76e59eb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-45.ec2.internal,UID:ip-10-0-134-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-45.ec2.internal,},FirstTimestamp:2026-09-17 19:05:56.898798059 +0000 UTC m=+2.064539327,LastTimestamp:2026-09-17 19:05:56.898798059 +0000 UTC m=+2.064539327,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-45.ec2.internal,}"
Sep 17 19:05:56.940487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.940471    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pkzlm"
Sep 17 19:05:56.940657 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.940583    2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-45.ec2.internal.18d63129a8e1a11b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-45.ec2.internal,UID:ip-10-0-134-45.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-45.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-45.ec2.internal,},FirstTimestamp:2026-09-17 19:05:56.923130139 +0000 UTC m=+2.088871414,LastTimestamp:2026-09-17 19:05:56.923130139 +0000 UTC m=+2.088871414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-45.ec2.internal,}"
Sep 17 19:05:56.950800 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.947897    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pkzlm"
Sep 17 19:05:56.963431 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963415    2586 manager.go:337] "Starting Device Plugin manager"
Sep 17 19:05:56.963504 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.963460    2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 19:05:56.963504 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963473    2586 server.go:87] "Starting device plugin registration server"
Sep 17 19:05:56.963795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963733    2586 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 19:05:56.963795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963757    2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 19:05:56.964116 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963883    2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 19:05:56.964116 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963950    2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 19:05:56.964116 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:56.963961    2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 19:05:56.964530 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.964508    2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 19:05:56.964610 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:56.964562    2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.008041 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.008013    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 19:05:57.009372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.009349    2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 19:05:57.009446 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.009399    2586 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 19:05:57.009446 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.009426    2586 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 19:05:57.009540 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.009482    2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 19:05:57.011636 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.011606    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:05:57.064080 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.064059    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:57.069848 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.069827    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:57.069940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.069862    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:57.069940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.069876    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:57.069940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.069904    2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.076422 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.076408    2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.076470 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.076430    2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-45.ec2.internal\": node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.091977 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.091960    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.109813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.109756    2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"]
Sep 17 19:05:57.109904 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.109826    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:57.111088 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.111073    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:57.111159 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.111100    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:57.111159 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.111109    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:57.112082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112070    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:57.112219 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112195    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.112271 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112227    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:57.112749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112736    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:57.112824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112755    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:57.112824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112741    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:57.112824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112764    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:57.112824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112790    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:57.112824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.112810    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:57.113817 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.113802    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.113876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.113824    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:57.114446 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.114422    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:57.114507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.114447    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:57.114507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.114471    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:57.144300 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.144284    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-45.ec2.internal\" not found" node="ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.148490 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.148476    2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-45.ec2.internal\" not found" node="ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.192045 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.192022    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.207082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.207060    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.207132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.207088    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.207132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.207103    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc7c99acaec5e04bdd3d6997df869ed9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-45.ec2.internal\" (UID: \"bc7c99acaec5e04bdd3d6997df869ed9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.292418 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.292394    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.307768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307746    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.307814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.307814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307788    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.307814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307811    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc7c99acaec5e04bdd3d6997df869ed9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-45.ec2.internal\" (UID: \"bc7c99acaec5e04bdd3d6997df869ed9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.307899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307845    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bc7c99acaec5e04bdd3d6997df869ed9-config\") pod \"kube-apiserver-proxy-ip-10-0-134-45.ec2.internal\" (UID: \"bc7c99acaec5e04bdd3d6997df869ed9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.307899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.307887    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59d476ce18ae64681e13745d1b849314-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal\" (UID: \"59d476ce18ae64681e13745d1b849314\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.393181 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.393093    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.447549 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.447521    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.451601 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.451583    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:57.494191 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.494172    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.594600 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.594574    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.695077 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.695020    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.795585 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.795562    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.863014 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.862992    2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 19:05:57.863503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.863115    2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 19:05:57.896456 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.896431    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:57.951741 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.951667    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 19:00:56 +0000 UTC" deadline="2028-03-17 11:57:58.122391223 +0000 UTC"
Sep 17 19:05:57.951741 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:57.951703    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13120h52m0.170691717s"
Sep 17 19:05:57.996970 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:57.996946    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:58.039326 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:58.039293    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7c99acaec5e04bdd3d6997df869ed9.slice/crio-94c6aa76566cea3410a5d052aba8689057590e67e0002628176241e0100e6865 WatchSource:0}: Error finding container 94c6aa76566cea3410a5d052aba8689057590e67e0002628176241e0100e6865: Status 404 returned error can't find the container with id 94c6aa76566cea3410a5d052aba8689057590e67e0002628176241e0100e6865
Sep 17 19:05:58.039803 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:05:58.039785    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d476ce18ae64681e13745d1b849314.slice/crio-fe7a5529210267e81bda16b5a8971d7f8299f45b002205358157b51c3a9667a1 WatchSource:0}: Error finding container fe7a5529210267e81bda16b5a8971d7f8299f45b002205358157b51c3a9667a1: Status 404 returned error can't find the container with id fe7a5529210267e81bda16b5a8971d7f8299f45b002205358157b51c3a9667a1
Sep 17 19:05:58.044498 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.044480    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:05:58.060669 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.060651    2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 19:05:58.061487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.061472    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 19:05:58.061554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.061500    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 19:05:58.061554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.061510    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeHasSufficientPID"
Sep 17 19:05:58.097856 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.097838    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:58.198359 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.198334    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:58.298890 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.298832    2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-45.ec2.internal\" not found"
Sep 17 19:05:58.372708 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.372686    2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:05:58.405833 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.405812    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:58.418419 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.418401    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:05:58.420065 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.420053    2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"
Sep 17 19:05:58.429316 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.429293    2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 19:05:58.458288 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.458270    2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:05:58.500985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.500961    2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:05:58.889022 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.888988    2586 apiserver.go:52] "Watching apiserver"
Sep 17 19:05:58.901701 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.901671    2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 19:05:58.902020 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.902001    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-fngdb","openshift-multus/multus-additional-cni-plugins-gqnlj","openshift-network-operator/iptables-alerter-8f4nq","openshift-ovn-kubernetes/ovnkube-node-6p2vh","kube-system/global-pull-secret-syncer-vnlqc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv","openshift-dns/node-resolver-9kxdj","openshift-image-registry/node-ca-2777q","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal","openshift-multus/multus-n7f8p","openshift-multus/network-metrics-daemon-m2gkt","openshift-network-diagnostics/network-check-target-m86xm","kube-system/konnectivity-agent-hnldj","kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal"]
Sep 17 19:05:58.903375 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.903353    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:58.904455 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.904434    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.905834    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.905863    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.905919    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9sd8p\""
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.906091    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.906239    2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.906529    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.906675    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 19:05:58.906814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.906678    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4hxkk\""
Sep 17 19:05:58.907286 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.907268    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:58.908238 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.908216    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.908494 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.908404    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 19:05:58.908494 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.908437    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.908684 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.908668    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-lgkqq\""
Sep 17 19:05:58.909342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.909324    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:58.909430 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.909398    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:05:58.909801 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.909772    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 19:05:58.909801 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.909789    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hcjfk\""
Sep 17 19:05:58.910578 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910553    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.910691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910580    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 19:05:58.910691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910644    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.910793 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910763    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 19:05:58.910850 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910810    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 19:05:58.910942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.910927    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:58.911665 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.911647    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:58.913195 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.913177    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:58.913534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.913516    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-kfxr6\""
Sep 17 19:05:58.914008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.913992    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.914193 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914179    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.914258 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914204    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.914345 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914329    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.914464 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914448    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 19:05:58.914539 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914523    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b74tn\""
Sep 17 19:05:58.914681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914668    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:58.914768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914750    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/102b7be6-ec29-407d-bb39-83b4f00acefb-iptables-alerter-script\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:58.914813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914784    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/102b7be6-ec29-407d-bb39-83b4f00acefb-host-slash\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:58.914864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a719325-0a92-413d-85d2-8848c0c0178f-hosts-file\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:58.914864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914835    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9r9g\" (UniqueName: \"kubernetes.io/projected/0a719325-0a92-413d-85d2-8848c0c0178f-kube-api-access-p9r9g\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:58.914864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914859    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb8d8e-7afe-467e-aa92-694eb841f26d-konnectivity-ca\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:58.915014 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914881    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqg2k\" (UniqueName: \"kubernetes.io/projected/102b7be6-ec29-407d-bb39-83b4f00acefb-kube-api-access-kqg2k\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:58.915014 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914905    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a719325-0a92-413d-85d2-8848c0c0178f-tmp-dir\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:58.915014 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.914926    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fdb8d8e-7afe-467e-aa92-694eb841f26d-agent-certs\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:58.915427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.915410    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.915841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.915421    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 19:05:58.915930 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.915541    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:58.917379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917359    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.917470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917400    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 19:05:58.917470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917451    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gh6hc\""
Sep 17 19:05:58.917577 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917563    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 19:05:58.917653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917594    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:58.917713 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.917672    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:05:58.917766 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917737    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ckq7n\""
Sep 17 19:05:58.917817 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917797    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 19:05:58.917865 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917819    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 19:05:58.917865 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.917861    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 19:05:58.918099 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.918082    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 19:05:58.918160 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.918084    2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 19:05:58.918160 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.918127    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rk6m\""
Sep 17 19:05:58.918744 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.918725    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:05:58.918829 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:58.918782    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:05:58.948499 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.948477    2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jlw2f"
Sep 17 19:05:58.958376 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:58.958356    2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jlw2f"
Sep 17 19:05:59.006738 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.006716    2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 19:05:59.013959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.013906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal" event={"ID":"59d476ce18ae64681e13745d1b849314","Type":"ContainerStarted","Data":"fe7a5529210267e81bda16b5a8971d7f8299f45b002205358157b51c3a9667a1"}
Sep 17 19:05:59.014961 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.014941    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal" event={"ID":"bc7c99acaec5e04bdd3d6997df869ed9","Type":"ContainerStarted","Data":"94c6aa76566cea3410a5d052aba8689057590e67e0002628176241e0100e6865"}
Sep 17 19:05:59.015071 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015054    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-socket-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.015167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015079    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-run\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-tmp\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015108    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-socket-dir-parent\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.015167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015156    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a719325-0a92-413d-85d2-8848c0c0178f-hosts-file\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015191    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb8d8e-7afe-467e-aa92-694eb841f26d-konnectivity-ca\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015225    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e054d15-f65f-4917-997c-85f1bf7d27eb-host\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015266    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-netns\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015283    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a719325-0a92-413d-85d2-8848c0c0178f-hosts-file\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015302    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-var-lib-kubelet\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015323    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvfh\" (UniqueName: \"kubernetes.io/projected/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-kube-api-access-ngvfh\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015364    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fdb8d8e-7afe-467e-aa92-694eb841f26d-agent-certs\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:59.015404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015387    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-device-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015411    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/e79c718f-09fc-4feb-8959-5e06fca44395-kube-api-access-w56g4\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015468    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e054d15-f65f-4917-997c-85f1bf7d27eb-serviceca\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-systemd-units\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015514    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-var-lib-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015535    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-bin\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015558    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-modprobe-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015603    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-host\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015644    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-multus-certs\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015711    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-env-overrides\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.015780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015776    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-kubelet\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015798    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-etc-kubernetes\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015797    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7fdb8d8e-7afe-467e-aa92-694eb841f26d-konnectivity-ca\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9r9g\" (UniqueName: \"kubernetes.io/projected/0a719325-0a92-413d-85d2-8848c0c0178f-kube-api-access-p9r9g\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015841    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b563966d-464c-4962-9c8e-71ba17af1708-kube-api-access-dmprk\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015866    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-sys\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015885    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-lib-modules\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015910    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-conf-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015927    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-daemon-config\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015944    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a719325-0a92-413d-85d2-8848c0c0178f-tmp-dir\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.015989    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-registration-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016032    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-kubelet\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016056    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-node-log\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016082    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b359f42-0dbc-4595-bf96-c67f951fb099-ovn-node-metrics-cert\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016104    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-script-lib\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016126    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-tuned\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016189    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqg2k\" (UniqueName: \"kubernetes.io/projected/102b7be6-ec29-407d-bb39-83b4f00acefb-kube-api-access-kqg2k\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016269    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-log-socket\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016293    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016316    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-netns\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016338    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k28wf\" (UniqueName: \"kubernetes.io/projected/dedcdb4b-2a47-461b-9b6a-f2067b550374-kube-api-access-k28wf\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016361    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92c8k\" (UniqueName: \"kubernetes.io/projected/0e054d15-f65f-4917-997c-85f1bf7d27eb-kube-api-access-92c8k\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016377    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-cnibin\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016392    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-config\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016431    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016454    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-bin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016341    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a719325-0a92-413d-85d2-8848c0c0178f-tmp-dir\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016525    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-multus\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.016770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-sys-fs\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c59z9\" (UniqueName: \"kubernetes.io/projected/55913654-3b16-495f-894f-400ad02b2485-kube-api-access-c59z9\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016575    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-netd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016588    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-hostroot\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016604    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/102b7be6-ec29-407d-bb39-83b4f00acefb-iptables-alerter-script\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016638    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016664    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-systemd\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016689    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016714    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-etc-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016661    2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016916    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-kubelet-config\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysconfig\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.016989    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/102b7be6-ec29-407d-bb39-83b4f00acefb-host-slash\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017015    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-etc-selinux\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017041    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-cni-binary-copy\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017068    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:05:59.017565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017093    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-slash\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017115    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-dbus\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017139    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-kubernetes\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017163    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-system-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017193    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-k8s-cni-cncf-io\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017217    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjlt8\" (UniqueName: \"kubernetes.io/projected/6b359f42-0dbc-4595-bf96-c67f951fb099-kube-api-access-mjlt8\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017246    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017264    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-system-cni-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017280    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-systemd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017306    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017340    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-conf\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017389    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-cnibin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017417    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017454    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-ovn\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017754    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/102b7be6-ec29-407d-bb39-83b4f00acefb-iptables-alerter-script\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.018272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.017774    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/102b7be6-ec29-407d-bb39-83b4f00acefb-host-slash\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.020650 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.020613    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7fdb8d8e-7afe-467e-aa92-694eb841f26d-agent-certs\") pod \"konnectivity-agent-hnldj\" (UID: \"7fdb8d8e-7afe-467e-aa92-694eb841f26d\") " pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:59.026109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.026082    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9r9g\" (UniqueName: \"kubernetes.io/projected/0a719325-0a92-413d-85d2-8848c0c0178f-kube-api-access-p9r9g\") pod \"node-resolver-9kxdj\" (UID: \"0a719325-0a92-413d-85d2-8848c0c0178f\") " pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.026362 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.026340    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqg2k\" (UniqueName: \"kubernetes.io/projected/102b7be6-ec29-407d-bb39-83b4f00acefb-kube-api-access-kqg2k\") pod \"iptables-alerter-8f4nq\" (UID: \"102b7be6-ec29-407d-bb39-83b4f00acefb\") " pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.117655 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117609    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b563966d-464c-4962-9c8e-71ba17af1708-kube-api-access-dmprk\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.117655 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117658    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-sys\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117674    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-lib-modules\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117691    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-conf-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117710    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-daemon-config\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117728    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-registration-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117742    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117749    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-sys\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117755    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-kubelet\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-kubelet\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117809    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-node-log\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b359f42-0dbc-4595-bf96-c67f951fb099-ovn-node-metrics-cert\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.117879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-script-lib\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117898    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-lib-modules\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-tuned\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117938    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117966    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.117981    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-log-socket\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118002    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118021    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-netns\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118035    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k28wf\" (UniqueName: \"kubernetes.io/projected/dedcdb4b-2a47-461b-9b6a-f2067b550374-kube-api-access-k28wf\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92c8k\" (UniqueName: \"kubernetes.io/projected/0e054d15-f65f-4917-997c-85f1bf7d27eb-kube-api-access-92c8k\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118070    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-cnibin\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118094    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118116    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-config\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118161    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-bin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118198    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-bin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118227    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-netns\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.118429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118270    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-multus\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118310    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-sys-fs\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118344    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c59z9\" (UniqueName: \"kubernetes.io/projected/55913654-3b16-495f-894f-400ad02b2485-kube-api-access-c59z9\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118375    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-netd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118399    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-hostroot\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118426    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118452    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-systemd\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118456    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-daemon-config\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.118556    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118749    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-sys-fs\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118768    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-netd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.118776    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:05:59.618692907 +0000 UTC m=+4.784434174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118829    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-hostroot\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118852    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-systemd\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118878    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118902    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-cni-multus\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118855    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.119398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118931    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-registration-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118869    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-conf-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118956    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-node-log\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.119021    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119029    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119024    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-etc-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.119087    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:05:59.619068882 +0000 UTC m=+4.784810150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119103    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-etc-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119113    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-kubelet-config\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysconfig\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119145    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119164    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-etc-selinux\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119195    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-cni-binary-copy\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119223    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119247    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-slash\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119249    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-kubelet-config\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119271    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-dbus\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.120290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119331    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-kubernetes\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-system-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119391    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dd1f444b-0e61-471b-9a77-6db913e39d7d-dbus\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.118800    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-cnibin\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-k8s-cni-cncf-io\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119472    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-system-cni-dir\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119493    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-kubernetes\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119514    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysconfig\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119496    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-config\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119560    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-slash\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119382    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-k8s-cni-cncf-io\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119568    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119602    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjlt8\" (UniqueName: \"kubernetes.io/projected/6b359f42-0dbc-4595-bf96-c67f951fb099-kube-api-access-mjlt8\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119644    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-etc-selinux\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119649    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.119682    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-ovn-kubernetes\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120549    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-system-cni-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.120965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120681    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-systemd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120751    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120858    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-tuned\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120891    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-conf\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120957    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-binary-copy\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.120999    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-cnibin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121123    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121152    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-system-cni-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121192    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-systemd\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121298    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-sysctl-conf\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dedcdb4b-2a47-461b-9b6a-f2067b550374-cni-binary-copy\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.121418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121406    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-ovnkube-script-lib\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.121749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121492    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-kubelet-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.121860 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121841    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-log-socket\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.121963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121943    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-cnibin\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.122018 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.121987    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.122069 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122025    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-ovn\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.122069 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122058    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-socket-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.122161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122091    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-run\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122115    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-tmp\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6b359f42-0dbc-4595-bf96-c67f951fb099-ovn-node-metrics-cert\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.122161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122152    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-socket-dir-parent\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.122359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e054d15-f65f-4917-997c-85f1bf7d27eb-host\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.122359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122227    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-netns\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.122359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122257    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-var-lib-kubelet\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvfh\" (UniqueName: \"kubernetes.io/projected/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-kube-api-access-ngvfh\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122334    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55913654-3b16-495f-894f-400ad02b2485-tuning-conf-dir\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.122590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122399    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-run-ovn\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.122590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-socket-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.122590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122544    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-run\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122653    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0e054d15-f65f-4917-997c-85f1bf7d27eb-host\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.122761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122712    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-run-netns\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.122856 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122794    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-var-lib-kubelet\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.122856 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122839    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-device-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.122952 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/e79c718f-09fc-4feb-8959-5e06fca44395-kube-api-access-w56g4\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.122952 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122909    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e054d15-f65f-4917-997c-85f1bf7d27eb-serviceca\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.122952 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122939    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-systemd-units\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.123091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122962    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-var-lib-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.123091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.122993    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-bin\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.123091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123025    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-modprobe-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.123091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123052    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-host\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.123091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123078    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-multus-certs\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.123327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123110    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.123327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123144    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-env-overrides\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.123327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-kubelet\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.123327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123217    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-etc-kubernetes\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.123327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123296    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-etc-kubernetes\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.123555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123346    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e79c718f-09fc-4feb-8959-5e06fca44395-device-dir\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.123722 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123703    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-multus-socket-dir-parent\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.123841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.123824    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-etc-modprobe-d\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.124175 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124156    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-var-lib-openvswitch\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.124263 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124205    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-host\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.124263 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124217    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-systemd-units\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.124263 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124226    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0e054d15-f65f-4917-997c-85f1bf7d27eb-serviceca\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.124414 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124351    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6b359f42-0dbc-4595-bf96-c67f951fb099-host-cni-bin\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.124414 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124406    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-run-multus-certs\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.124634 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124599    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/55913654-3b16-495f-894f-400ad02b2485-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.124703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124635    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dedcdb4b-2a47-461b-9b6a-f2067b550374-host-var-lib-kubelet\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.124919 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.124901    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6b359f42-0dbc-4595-bf96-c67f951fb099-env-overrides\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.128306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.128274    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-tmp\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.129709 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.129482    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmprk\" (UniqueName: \"kubernetes.io/projected/b563966d-464c-4962-9c8e-71ba17af1708-kube-api-access-dmprk\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.129709 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.129670    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c59z9\" (UniqueName: \"kubernetes.io/projected/55913654-3b16-495f-894f-400ad02b2485-kube-api-access-c59z9\") pod \"multus-additional-cni-plugins-gqnlj\" (UID: \"55913654-3b16-495f-894f-400ad02b2485\") " pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.129864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.129817    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k28wf\" (UniqueName: \"kubernetes.io/projected/dedcdb4b-2a47-461b-9b6a-f2067b550374-kube-api-access-k28wf\") pod \"multus-n7f8p\" (UID: \"dedcdb4b-2a47-461b-9b6a-f2067b550374\") " pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.129864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.129860    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92c8k\" (UniqueName: \"kubernetes.io/projected/0e054d15-f65f-4917-997c-85f1bf7d27eb-kube-api-access-92c8k\") pod \"node-ca-2777q\" (UID: \"0e054d15-f65f-4917-997c-85f1bf7d27eb\") " pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.131071 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.131054    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjlt8\" (UniqueName: \"kubernetes.io/projected/6b359f42-0dbc-4595-bf96-c67f951fb099-kube-api-access-mjlt8\") pod \"ovnkube-node-6p2vh\" (UID: \"6b359f42-0dbc-4595-bf96-c67f951fb099\") " pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.131544 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.131529    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvfh\" (UniqueName: \"kubernetes.io/projected/176fa980-8c40-4a9d-b8e0-acdbcb5085dc-kube-api-access-ngvfh\") pod \"tuned-fngdb\" (UID: \"176fa980-8c40-4a9d-b8e0-acdbcb5085dc\") " pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.131828 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.131813    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:05:59.131828 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.131830    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:05:59.131918 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.131839    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:05:59.131918 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.131886    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:05:59.63187274 +0000 UTC m=+4.797614012 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:05:59.133197 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.133179    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w56g4\" (UniqueName: \"kubernetes.io/projected/e79c718f-09fc-4feb-8959-5e06fca44395-kube-api-access-w56g4\") pod \"aws-ebs-csi-driver-node-p84hv\" (UID: \"e79c718f-09fc-4feb-8959-5e06fca44395\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.216243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.216179    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9kxdj"
Sep 17 19:05:59.223588 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.223563    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:05:59.232156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.232129    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8f4nq"
Sep 17 19:05:59.238773 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.238757    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:05:59.247199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.247175    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv"
Sep 17 19:05:59.253724 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.253708    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2777q"
Sep 17 19:05:59.261444 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.261319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fngdb"
Sep 17 19:05:59.268681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.268663    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-gqnlj"
Sep 17 19:05:59.275246 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.275232    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n7f8p"
Sep 17 19:05:59.375919 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.375889    2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 19:05:59.625760 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.625687    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:05:59.625760 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.625724    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:05:59.625955 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.625837    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:05:59.625955 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.625843    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:05:59.625955 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.625894    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:00.625879466 +0000 UTC m=+5.791620718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:05:59.625955 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.625912    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:00.625906307 +0000 UTC m=+5.791647559 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:05:59.726637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.726592    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:05:59.726768 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.726736    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:05:59.726768 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.726755    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:05:59.726768 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.726764    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:05:59.726878 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:05:59.726812    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:00.726798036 +0000 UTC m=+5.892539289 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:05:59.960355 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.960268    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:00:58 +0000 UTC" deadline="2028-06-25 20:12:20.163884084 +0000 UTC"
Sep 17 19:05:59.960355 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:05:59.960308    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15529h6m20.203579862s"
Sep 17 19:06:00.058827 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:00.058798    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a719325_0a92_413d_85d2_8848c0c0178f.slice/crio-e2dadbf023ca3b15218833722423b3bd67ee5376d537f9c47d6fd7ed45c6db40 WatchSource:0}: Error finding container e2dadbf023ca3b15218833722423b3bd67ee5376d537f9c47d6fd7ed45c6db40: Status 404 returned error can't find the container with id e2dadbf023ca3b15218833722423b3bd67ee5376d537f9c47d6fd7ed45c6db40
Sep 17 19:06:00.060719 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:00.060698    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b359f42_0dbc_4595_bf96_c67f951fb099.slice/crio-8c1f17ac7f201a38e6ec10d8b65124d02dce0427ee11d8091abe5dd60bd28a80 WatchSource:0}: Error finding container 8c1f17ac7f201a38e6ec10d8b65124d02dce0427ee11d8091abe5dd60bd28a80: Status 404 returned error can't find the container with id 8c1f17ac7f201a38e6ec10d8b65124d02dce0427ee11d8091abe5dd60bd28a80
Sep 17 19:06:00.069605 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:00.069293    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fdb8d8e_7afe_467e_aa92_694eb841f26d.slice/crio-38756dbef1bc9c93d793a6203089910045068aaca17d3851e16ffc1bb2c20cda WatchSource:0}: Error finding container 38756dbef1bc9c93d793a6203089910045068aaca17d3851e16ffc1bb2c20cda: Status 404 returned error can't find the container with id 38756dbef1bc9c93d793a6203089910045068aaca17d3851e16ffc1bb2c20cda
Sep 17 19:06:00.072938 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:00.071599    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod102b7be6_ec29_407d_bb39_83b4f00acefb.slice/crio-bf573ee6fd222f9ffe4bc93cfe71f19641525d2db1ab419a8534c5d3a59af14e WatchSource:0}: Error finding container bf573ee6fd222f9ffe4bc93cfe71f19641525d2db1ab419a8534c5d3a59af14e: Status 404 returned error can't find the container with id bf573ee6fd222f9ffe4bc93cfe71f19641525d2db1ab419a8534c5d3a59af14e
Sep 17 19:06:00.633104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:00.632856    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:00.633284 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.633024    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:00.633284 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:00.633139    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:00.633284 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.633207    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:02.633186498 +0000 UTC m=+7.798927756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:00.633284 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.633261    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:00.633510 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.633305    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:02.633293238 +0000 UTC m=+7.799034492 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:00.733943 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:00.733907    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:00.734116 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.734091    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:00.734116 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.734116    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:00.734306 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.734129    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:00.734306 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:00.734194    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:02.734175901 +0000 UTC m=+7.899917179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:00.960512 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:00.960427    2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 19:00:58 +0000 UTC" deadline="2028-06-08 09:19:46.203011414 +0000 UTC"
Sep 17 19:06:00.960512 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:00.960466    2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15110h13m45.242550248s"
Sep 17 19:06:01.012426 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.012397    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:01.012584 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:01.012535    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:01.012677 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.012646    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:01.012743 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:01.012722    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:01.012817 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.012804    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:01.013212 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:01.012887    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:01.050968 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.050934    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8f4nq" event={"ID":"102b7be6-ec29-407d-bb39-83b4f00acefb","Type":"ContainerStarted","Data":"bf573ee6fd222f9ffe4bc93cfe71f19641525d2db1ab419a8534c5d3a59af14e"}
Sep 17 19:06:01.058514 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.058486    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hnldj" event={"ID":"7fdb8d8e-7afe-467e-aa92-694eb841f26d","Type":"ContainerStarted","Data":"38756dbef1bc9c93d793a6203089910045068aaca17d3851e16ffc1bb2c20cda"}
Sep 17 19:06:01.064481 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.064428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7f8p" event={"ID":"dedcdb4b-2a47-461b-9b6a-f2067b550374","Type":"ContainerStarted","Data":"29b182a0e639c5a2e4408e34c2382c606d68cfb4cda34efb9e568a09ca111001"}
Sep 17 19:06:01.067806 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.067781    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"8c1f17ac7f201a38e6ec10d8b65124d02dce0427ee11d8091abe5dd60bd28a80"}
Sep 17 19:06:01.088861 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.088834    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9kxdj" event={"ID":"0a719325-0a92-413d-85d2-8848c0c0178f","Type":"ContainerStarted","Data":"e2dadbf023ca3b15218833722423b3bd67ee5376d537f9c47d6fd7ed45c6db40"}
Sep 17 19:06:01.103640 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.103566    2586 generic.go:356] "Generic (PLEG): container finished" podID="59d476ce18ae64681e13745d1b849314" containerID="5b725d9a79e0c4fba35e8fd5befc8e1fe25a8feb45b055506fe88105317ef9a6" exitCode=0
Sep 17 19:06:01.103753 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.103674    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal" event={"ID":"59d476ce18ae64681e13745d1b849314","Type":"ContainerDied","Data":"5b725d9a79e0c4fba35e8fd5befc8e1fe25a8feb45b055506fe88105317ef9a6"}
Sep 17 19:06:01.119731 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.119580    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv" event={"ID":"e79c718f-09fc-4feb-8959-5e06fca44395","Type":"ContainerStarted","Data":"e06fc06c1089ea78c8e34eb100c403b6d21ee2d9c76058ea1f90aff3d5e0027d"}
Sep 17 19:06:01.138385 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.137416    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fngdb" event={"ID":"176fa980-8c40-4a9d-b8e0-acdbcb5085dc","Type":"ContainerStarted","Data":"4f2cb0a2bbd6752759685827436183adce493ad6968373cb7bfd8d97bc4bba2f"}
Sep 17 19:06:01.153700 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.153069    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal" event={"ID":"bc7c99acaec5e04bdd3d6997df869ed9","Type":"ContainerStarted","Data":"b919e5f2c44d1fa00df90e3afab37e35a74f4c68897c5828f180fff899a8a021"}
Sep 17 19:06:01.165319 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.165270    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2777q" event={"ID":"0e054d15-f65f-4917-997c-85f1bf7d27eb","Type":"ContainerStarted","Data":"5c7dd0da26ba87360bea462d82dd14d26cac2d868084171c07b40a97d801b42c"}
Sep 17 19:06:01.170538 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.169980    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-45.ec2.internal" podStartSLOduration=3.1699634740000002 podStartE2EDuration="3.169963474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:01.169739332 +0000 UTC m=+6.335480608" watchObservedRunningTime="2026-09-17 19:06:01.169963474 +0000 UTC m=+6.335704752"
Sep 17 19:06:01.175385 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:01.175359    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerStarted","Data":"3a83dac0992e3e33cac09ff46e3a9451ae2db9f5f8932aa07f0cc524e8512c78"}
Sep 17 19:06:02.194668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:02.194615    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal" event={"ID":"59d476ce18ae64681e13745d1b849314","Type":"ContainerStarted","Data":"c85a0c4a51ab2e9410cf986aa68ef0088b94c0182da4639416f06452af45dfde"}
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:02.651875    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:02.651928    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.652110    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.652182    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:06.652154637 +0000 UTC m=+11.817895909 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.652570    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:02.652673 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.652639    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:06.652606214 +0000 UTC m=+11.818347471 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:02.752943 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:02.752906    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:02.753123 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.753088    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:02.753123 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.753106    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:02.753123 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.753118    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:02.753287 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:02.753177    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:06.753157688 +0000 UTC m=+11.918899128 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:03.013146 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:03.013072    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:03.013299 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:03.013212    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:03.013375 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:03.013318    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:03.013440 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:03.013416    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:03.013497 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:03.013484    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:03.013721 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:03.013551    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:05.011771    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:05.011881    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:05.012200    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:05.012284    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:05.012330    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:05.012537 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:05.012414    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:06.685087    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:06.685143    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.685300    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.685360    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.685340974 +0000 UTC m=+19.851082241 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.685420    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:06.685817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.685450    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.685438868 +0000 UTC m=+19.851180135 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:06.786016 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:06.785978    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:06.786192 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.786149    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:06.786192 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.786170    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:06.786192 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.786182    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:06.786360 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:06.786237    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:14.786219971 +0000 UTC m=+19.951961230 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:07.011638 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:07.011133    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:07.011638 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:07.011249    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:07.011638 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:07.011500    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:07.012013 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:07.011640    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:07.012013 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:07.011698    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:07.012013 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:07.011807    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:09.011743    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:09.011862    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:09.012233    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:09.012343    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:09.012385    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:09.012501 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:09.012446    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:11.010124 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:11.010082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:11.010576 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:11.010082    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:11.010576 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:11.010214    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:11.010576 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:11.010229    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:11.010576 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:11.010356    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:11.010576 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:11.010453    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:13.010257 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:13.010223    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:13.010750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:13.010227    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:13.010750 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:13.010371    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:13.010750 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:13.010433    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:13.010750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:13.010223    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:13.010750 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:13.010523    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:14.746386 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:14.746357    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:14.746386 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:14.746392    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:14.746786 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.746511    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:14.746786 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.746512    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:14.746786 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.746560    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:06:30.746546727 +0000 UTC m=+35.912287980 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:14.746786 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.746573    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:30.746566745 +0000 UTC m=+35.912307999 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:14.847100 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:14.847048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:14.847279 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.847257    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:14.847347 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.847280    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:14.847347 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.847294    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:14.847436 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:14.847362    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:06:30.847341801 +0000 UTC m=+36.013083073 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:15.009913    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:15.009912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:15.010050    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:15.010047    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:15.010161    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:15.010448 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:15.010247    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:17.010801 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:17.010762    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:17.011242 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:17.010821    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:17.011242 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:17.010902    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:17.011242 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:17.010905    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:17.011242 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:17.010982    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:17.011242 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:17.011074    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:18.225214 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.225007    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2777q" event={"ID":"0e054d15-f65f-4917-997c-85f1bf7d27eb","Type":"ContainerStarted","Data":"ed7f87f2c0067208f67a7046e718b2a5c09016c6ca99c764044d61f8db8ba2ac"}
Sep 17 19:06:18.226448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.226426    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="20ad304051fb33aad57dead492447527b0d833253433095f439dc37891e6596e" exitCode=0
Sep 17 19:06:18.226524 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.226462    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"20ad304051fb33aad57dead492447527b0d833253433095f439dc37891e6596e"}
Sep 17 19:06:18.228119 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.228092    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hnldj" event={"ID":"7fdb8d8e-7afe-467e-aa92-694eb841f26d","Type":"ContainerStarted","Data":"0e18084d496643314cad831307495238e4540632d8177592f2d677248a8230ee"}
Sep 17 19:06:18.229559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.229526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n7f8p" event={"ID":"dedcdb4b-2a47-461b-9b6a-f2067b550374","Type":"ContainerStarted","Data":"f5355c8efd4d20d520882da63a5d5dfc1e14ddec42a747c2517ede71711f88d6"}
Sep 17 19:06:18.233935 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.233919    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:06:18.234261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.234242    2586 generic.go:356] "Generic (PLEG): container finished" podID="6b359f42-0dbc-4595-bf96-c67f951fb099" containerID="a3f33e7000e17067d101058a5848cd74c7406dc2e1db93f4a455f8c5f850a353" exitCode=1
Sep 17 19:06:18.234330 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.234306    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"d5d7169fb099ddd0ce1945b7a51e5c6d1fce6852cf2318055dede931ed5712de"}
Sep 17 19:06:18.234371 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.234358    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerDied","Data":"a3f33e7000e17067d101058a5848cd74c7406dc2e1db93f4a455f8c5f850a353"}
Sep 17 19:06:18.234411 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.234388    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"c4f57e63d3f635456984b1416445c15420ca82d146f41fc7733167c5ce67e93f"}
Sep 17 19:06:18.235564 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.235548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9kxdj" event={"ID":"0a719325-0a92-413d-85d2-8848c0c0178f","Type":"ContainerStarted","Data":"42572dfe03908c8c267b0b2c2e286cb1f5d99685f4ec3eeaf5cf6c4fc9e41ae2"}
Sep 17 19:06:18.236972 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.236947    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv" event={"ID":"e79c718f-09fc-4feb-8959-5e06fca44395","Type":"ContainerStarted","Data":"c15296218e9ea48385e5ac91b5c32a8f8bc7e505c40ad66fcf00491d5bf0b20a"}
Sep 17 19:06:18.238570 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.238548    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fngdb" event={"ID":"176fa980-8c40-4a9d-b8e0-acdbcb5085dc","Type":"ContainerStarted","Data":"bc133e4eff1c5929f28ede51169bfd83c6475b38c161a0a3709e0548db918749"}
Sep 17 19:06:18.249808 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.249774    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-45.ec2.internal" podStartSLOduration=20.249763038 podStartE2EDuration="20.249763038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:02.212204818 +0000 UTC m=+7.377946113" watchObservedRunningTime="2026-09-17 19:06:18.249763038 +0000 UTC m=+23.415504314"
Sep 17 19:06:18.271327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.271287    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2777q" podStartSLOduration=3.759751324 podStartE2EDuration="21.271274207s" totalImagesPullingTime="17.511522883s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:18.250141185 +0000 UTC m=+23.415882480" watchObservedRunningTime="2026-09-17 19:06:18.271274207 +0000 UTC m=+23.437015482"
Sep 17 19:06:18.299658 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.299606    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hnldj" podStartSLOduration=3.784822964 podStartE2EDuration="21.29959431s" totalImagesPullingTime="17.514771346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:18.271734947 +0000 UTC m=+23.437476234" watchObservedRunningTime="2026-09-17 19:06:18.29959431 +0000 UTC m=+23.465335588"
Sep 17 19:06:18.299728 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.299709    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fngdb" podStartSLOduration=3.780989535 podStartE2EDuration="21.299704277s" totalImagesPullingTime="17.518714742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:18.299190918 +0000 UTC m=+23.464932194" watchObservedRunningTime="2026-09-17 19:06:18.299704277 +0000 UTC m=+23.465445548"
Sep 17 19:06:18.348158 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.348088    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-9kxdj" podStartSLOduration=3.816421006 podStartE2EDuration="21.348074874s" totalImagesPullingTime="17.531653868s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:18.347889968 +0000 UTC m=+23.513631264" watchObservedRunningTime="2026-09-17 19:06:18.348074874 +0000 UTC m=+23.513816149"
Sep 17 19:06:18.355373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.355346    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:18.371912 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.371873    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n7f8p" podStartSLOduration=3.761151153 podStartE2EDuration="21.37185611s" totalImagesPullingTime="17.610704957s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:18.371074574 +0000 UTC m=+23.536815869" watchObservedRunningTime="2026-09-17 19:06:18.37185611 +0000 UTC m=+23.537597385"
Sep 17 19:06:18.426980 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.426927    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:18.527605 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.527574    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:18.627281 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.627256    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:18.684028 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:18.683997    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:19.010018 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.009993    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:19.010153 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.010009    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:19.010222 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:19.010143    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:19.010222 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:19.010194    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:19.010337 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.010302    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:19.010404 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:19.010375    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:19.142142 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.142121    2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 19:06:19.243600 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.243562    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8f4nq" event={"ID":"102b7be6-ec29-407d-bb39-83b4f00acefb","Type":"ContainerStarted","Data":"611d4eb69e1625fef528d65ac037ec1635a4bf61b55db250164626d3e343a47c"}
Sep 17 19:06:19.247710 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.247683    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:06:19.248132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.248111    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"47aa6d5bf0392b38e85dee277c408f707560f7d6a6d6eb09a2f046bfbc4ab01f"}
Sep 17 19:06:19.248243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.248183    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"70281c71364ab0789b68373510317c5088628d1efd951452b0b198a374b6116a"}
Sep 17 19:06:19.248243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.248230    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"a73a0f449e561678eaa2bcdfa4a1451eb2cb2d0bdfabf29bda467241d21673a7"}
Sep 17 19:06:19.249982 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.249960    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv" event={"ID":"e79c718f-09fc-4feb-8959-5e06fca44395","Type":"ContainerStarted","Data":"7d931bca680726cfe4367feb0ba00e07555f0ea749976c12ea991606ebce99ec"}
Sep 17 19:06:19.392924 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.392895    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:19.449938 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.449903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:19.657306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.656988    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:06:19.657719 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.657685    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:06:19.674387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.673602    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8f4nq" podStartSLOduration=5.162137424 podStartE2EDuration="22.673582324s" totalImagesPullingTime="17.5114449s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:19.262075267 +0000 UTC m=+24.427816561" watchObservedRunningTime="2026-09-17 19:06:19.673582324 +0000 UTC m=+24.839323600"
Sep 17 19:06:19.976594 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.976438    2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T19:06:19.142136785Z","UUID":"e80e563f-a8a3-4054-b59b-4263440dc573","Handler":null,"Name":"","Endpoint":""}
Sep 17 19:06:19.978003 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.977979    2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 19:06:19.978139 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:19.978012    2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 19:06:20.252470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:20.252397    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:06:20.252909 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:20.252686    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hnldj"
Sep 17 19:06:20.335262 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:20.335237    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:20.519302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:20.519138    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:20.618447 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:20.618423    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:21.010446 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.010418    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:21.010649 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.010419    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:21.010649 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.010555    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:21.010649 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:21.010636    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:21.010817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:21.010528    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:21.010817 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:21.010716    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:21.258110 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.258083    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:06:21.258577 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.258502    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"440f630351ebf05b7ea32e1c662dcf98db9eab68c216bf954e8e807e1620397c"}
Sep 17 19:06:21.260995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.260920    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv" event={"ID":"e79c718f-09fc-4feb-8959-5e06fca44395","Type":"ContainerStarted","Data":"4bbfdf98707a12d8b84284652f2b8a99538ae5e39716a39e1c32439e96b8aafb"}
Sep 17 19:06:21.282684 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:21.282645    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-p84hv" podStartSLOduration=4.466112304 podStartE2EDuration="24.282613292s" totalImagesPullingTime="19.816500988s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:21.282390728 +0000 UTC m=+26.448132024" watchObservedRunningTime="2026-09-17 19:06:21.282613292 +0000 UTC m=+26.448354569"
Sep 17 19:06:22.054258 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:22.054223    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:22.115108 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:22.115076    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.010637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.010492    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:23.010637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.010531    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:23.010637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.010608    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:23.011247 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:23.010650    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:23.011247 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:23.010722    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:23.011247 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:23.010785    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:23.130388 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.130349    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.213212 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.213176    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.280115 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.280079    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.379053 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.379022    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.626196 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.626043    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:23.965137 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:23.965034    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:24.064491 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.064462    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:24.270533 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.270468    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:06:24.270895 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.270873    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"f450fcc20c6f6c40e31c5f5fe049c21a710e3e46181f4fed9c19faa723fdc20f"}
Sep 17 19:06:24.271142 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.271106    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:24.271265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.271179    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:24.271265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.271227    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:24.271396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.271265    2586 scope.go:117] "RemoveContainer" containerID="a3f33e7000e17067d101058a5848cd74c7406dc2e1db93f4a455f8c5f850a353"
Sep 17 19:06:24.273054 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.273029    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="03281d135084bb1af49e5da72e4961eb801ce84dbfc0604f3dc83c4e2a193055" exitCode=0
Sep 17 19:06:24.273162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.273064    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"03281d135084bb1af49e5da72e4961eb801ce84dbfc0604f3dc83c4e2a193055"}
Sep 17 19:06:24.288519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.288502    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:24.288600 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.288591    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:24.363917 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.363890    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:24.423986 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:24.423954    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:25.010350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.010324    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:25.010443 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.010392    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:25.010443 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.010412    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:25.010563 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.010515    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:25.010672 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.010579    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:25.010730 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.010682    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:25.197536 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.197504    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:25.276960 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.276891    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="754a7da8ea8b023c4985e6bebe30a1151fcc72ffdbc881183aa7afa4c81e3f42" exitCode=0
Sep 17 19:06:25.277093 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.276973    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"754a7da8ea8b023c4985e6bebe30a1151fcc72ffdbc881183aa7afa4c81e3f42"}
Sep 17 19:06:25.281025 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.281009    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:06:25.281424 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.281402    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" event={"ID":"6b359f42-0dbc-4595-bf96-c67f951fb099","Type":"ContainerStarted","Data":"66a7ef3992f2a1c3934e3bf3337ad4a69b8ff71ae08203223a66b1489b425cce"}
Sep 17 19:06:25.297089 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.297063    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:25.340473 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.340437    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh" podStartSLOduration=10.535750935 podStartE2EDuration="28.340423995s" totalImagesPullingTime="17.80467306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:25.33889255 +0000 UTC m=+30.504633840" watchObservedRunningTime="2026-09-17 19:06:25.340423995 +0000 UTC m=+30.506165270"
Sep 17 19:06:25.356082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.356058    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m86xm"]
Sep 17 19:06:25.356179 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.356145    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:25.356282 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.356264    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:25.356598 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.356573    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2gkt"]
Sep 17 19:06:25.356674 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.356657    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:25.356744 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.356729    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:25.367479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.367458    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnlqc"]
Sep 17 19:06:25.367559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.367542    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:25.367671 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:25.367652    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:25.570466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.570420    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:25.670544 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:25.670501    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.361939 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.361899    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.463266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.463236    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.562774 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.562744    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.667300 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.667078    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.764695 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.764673    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:26.989187 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:26.989111    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:27.010923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.010897    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:27.011049 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:27.010989    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:27.011049 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.011006    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:27.011114 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.011059    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:27.011146 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:27.011120    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:27.011235 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:27.011209    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:27.080639 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.080606    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:27.140691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.140666    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:27.240585 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.240520    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:27.289154 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.289128    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="709fe43d65ecb2c244af39aa4242703cc2708c1c5b239aeb13a8cdef2c704fb2" exitCode=0
Sep 17 19:06:27.289293 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.289157    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"709fe43d65ecb2c244af39aa4242703cc2708c1c5b239aeb13a8cdef2c704fb2"}
Sep 17 19:06:27.320245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.320223    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:27.637826 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:27.637804    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9kxdj_0a719325-0a92-413d-85d2-8848c0c0178f/dns-node-resolver/0.log"
Sep 17 19:06:28.007598 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.007548    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.108130 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.108101    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.271020 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.270945    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.328809 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.328767    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.385194 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.385166    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.632381 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.632352    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2777q_0e054d15-f65f-4917-997c-85f1bf7d27eb/node-ca/0.log"
Sep 17 19:06:28.885993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.885879    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:28.951017 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:28.950987    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:29.009691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:29.009664    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:29.009691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:29.009679    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:29.009911 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:29.009671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:29.009911 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:29.009774    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:29.009911 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:29.009829    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:29.009911 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:29.009887    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:29.837653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:29.837606    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:30.771122 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:30.771089    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:30.771690 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:30.771133    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:30.771690 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.771234    2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:30.771690 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.771254    2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:30.771690 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.771311    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs podName:b563966d-464c-4962-9c8e-71ba17af1708 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.771289077 +0000 UTC m=+67.937030339 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs") pod "network-metrics-daemon-m2gkt" (UID: "b563966d-464c-4962-9c8e-71ba17af1708") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 19:06:30.771690 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.771331    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret podName:dd1f444b-0e61-471b-9a77-6db913e39d7d nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.771321218 +0000 UTC m=+67.937062477 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret") pod "global-pull-secret-syncer-vnlqc" (UID: "dd1f444b-0e61-471b-9a77-6db913e39d7d") : object "kube-system"/"original-pull-secret" not registered
Sep 17 19:06:30.871988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:30.871952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:30.872167 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.872142    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 19:06:30.872227 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.872173    2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 19:06:30.872227 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.872187    2586 projected.go:196] Error preparing data for projected volume kube-api-access-8mr9t for pod openshift-network-diagnostics/network-check-target-m86xm: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:30.872319 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:30.872251    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t podName:3d798031-06ea-48a0-90e7-f4c6928307d1 nodeName:}" failed. No retries permitted until 2026-09-17 19:07:02.872231839 +0000 UTC m=+68.037973117 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8mr9t" (UniqueName: "kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t") pod "network-check-target-m86xm" (UID: "3d798031-06ea-48a0-90e7-f4c6928307d1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 19:06:31.010569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.010480    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:31.010569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.010480    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:31.010811 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:31.010603    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:31.010811 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.010482    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:31.010811 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:31.010730    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:31.010811 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:31.010804    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:31.169474 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.169437    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.400755 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.400717    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.491997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.491967    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.555858 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.555824    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.617900 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.617864    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.729262 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.729187    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:31.861426 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:31.861391    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:32.461123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:32.461096    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:32.890215 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:32.890186    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:33.009970 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.009943    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:33.010097 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.009945    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:33.010097 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.010058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:33.010168 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:33.010030    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:33.010168 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:33.010123    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:33.010246 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:33.010174    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:33.125280 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.125254    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:33.280371 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.280348    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:33.304566 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.304546    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="54880de44818fe0c2c0f7ba87f48c55f95034f7aab319b925bee9d22a5dea6e2" exitCode=0
Sep 17 19:06:33.304681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:33.304574    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"54880de44818fe0c2c0f7ba87f48c55f95034f7aab319b925bee9d22a5dea6e2"}
Sep 17 19:06:34.065869 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.065835    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:34.276675 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.276597    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:34.309596 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.309568    2586 generic.go:356] "Generic (PLEG): container finished" podID="55913654-3b16-495f-894f-400ad02b2485" containerID="5e3e069271fa24c5515ecc2c50c43b78b21dba44f72b6e05e432242bac78f39a" exitCode=0
Sep 17 19:06:34.309750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.309651    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerDied","Data":"5e3e069271fa24c5515ecc2c50c43b78b21dba44f72b6e05e432242bac78f39a"}
Sep 17 19:06:34.663975 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.663906    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:34.853055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.853024    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:34.916657 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:34.916575    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:35.009951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.009916    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:35.010080 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.009932    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:35.010080 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:35.010015    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:35.010080 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.010025    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:35.010247 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:35.010110    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:35.010247 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:35.010173    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:35.315959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.315781    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" event={"ID":"55913654-3b16-495f-894f-400ad02b2485","Type":"ContainerStarted","Data":"dc677a1023f49bec307aca40712104ec8676488b943ea8a7a1515e5ae516cca9"}
Sep 17 19:06:35.340307 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.340269    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-gqnlj" podStartSLOduration=8.518857456 podStartE2EDuration="38.340252177s" totalImagesPullingTime="29.821394721s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:35.338959373 +0000 UTC m=+40.504700661" watchObservedRunningTime="2026-09-17 19:06:35.340252177 +0000 UTC m=+40.505993454"
Sep 17 19:06:35.791928 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.791906    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:35.891887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.891854    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:35.990408 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:35.990382    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.090173 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.090100    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.256855 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.256818    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.309391 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.309369    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.664261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.664232    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.763928 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.763901    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:36.864760 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:36.864733    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.011554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.011490    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:37.011714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.011604    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:37.011714 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:37.011659    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:37.011714 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:37.011697    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:37.011982 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.011731    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:37.011982 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:37.011806    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:37.058014 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.057984    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.115004 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.114975    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.280802 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.280769    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.673963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.673894    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.781934 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.781903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.872509 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.872486    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:37.925493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:37.925444    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.045667 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.045644    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.101386 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.101363    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.225123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.225059    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.286362 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.286341    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.459635 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.459608    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.648439 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.648408    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:38.733151 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:38.733126    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.010668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.010606    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:39.010791 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.010607    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:39.010791 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:39.010709    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:39.010871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.010608    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:39.010871 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:39.010841    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:39.010871 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:39.010788    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:39.261711 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.261643    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.440507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.440478    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.498942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.498903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.642963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.642935    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.836216 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.836186    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:39.888210 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:39.888184    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:40.060294 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:40.060274    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:40.121830 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:40.121809    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:40.477383 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:40.477313    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:41.010656 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.010611    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:41.010656 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.010652    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:41.011177 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.010615    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:41.011177 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:41.010737    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:41.011177 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:41.010791    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:41.011177 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:41.010860    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:41.063905 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.063883    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:41.655047 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.655023    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:41.862758 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.862729    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:41.987752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:41.987678    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:42.090030 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:42.090002    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:42.253998 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:42.253935    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:43.010553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:43.010499    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:43.010553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:43.010510    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:43.010733 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:43.010502    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:43.010733 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:43.010590    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:43.010733 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:43.010711    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:43.010887 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:43.010854    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:45.010510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:45.010472    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:45.010510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:45.010492    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:45.010987 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:45.010482    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:45.010987 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:45.010570    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:45.010987 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:45.010670    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:45.010987 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:45.010731    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:45.455296 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:45.455259    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:45.560306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:45.560279    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:47.010594 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:47.010558    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:47.011132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:47.010673    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:47.011132 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:47.010674    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:47.011132 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:47.010818    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:47.011132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:47.010852    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:47.011132 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:47.010926    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:49.010107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.010081    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:49.010107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.010103    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:49.010499 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.010103    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:49.010499 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:49.010171    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-m2gkt" podUID="b563966d-464c-4962-9c8e-71ba17af1708"
Sep 17 19:06:49.010499 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:49.010275    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnlqc" podUID="dd1f444b-0e61-471b-9a77-6db913e39d7d"
Sep 17 19:06:49.010499 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:49.010338    2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-m86xm" podUID="3d798031-06ea-48a0-90e7-f4c6928307d1"
Sep 17 19:06:49.436953 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.436921    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.521955 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.521922    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.618532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.618508    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.719505 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.719443    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.915718 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.915686    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:49.975344 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:49.975275    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:50.600400 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.600327    2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-45.ec2.internal" event="NodeReady"
Sep 17 19:06:50.600756 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.600423    2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 19:06:50.664193 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.664165    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74c559dfd4-pcvpj"]
Sep 17 19:06:50.685290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.685265    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.690117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.690098    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zrgzz"]
Sep 17 19:06:50.695263 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.695245    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 19:06:50.695354 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.695327    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 19:06:50.695354 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.695335    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 19:06:50.695579 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.695562    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-6d4gz\""
Sep 17 19:06:50.705436 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.705417    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.706096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.706077    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 19:06:50.709418 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.709399    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c559dfd4-pcvpj"]
Sep 17 19:06:50.710864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.710846    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4lr2c"]
Sep 17 19:06:50.714995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.714978    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90fcbb05-90ab-44ec-b698-89859beb174f-ca-trust-extracted\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715018    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbc8v\" (UniqueName: \"kubernetes.io/projected/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-kube-api-access-zbc8v\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.715075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715034    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-bound-sa-token\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715054    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqcnj\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-kube-api-access-cqcnj\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715087    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-image-registry-private-configuration\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715124    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-tmp-dir\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.715174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715153    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-trusted-ca\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715171    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-config-volume\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.715289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715185    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-metrics-tls\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.715289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715212    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-registry-tls\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-registry-certificates\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715263    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-installation-pull-secrets\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.715534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715517    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 19:06:50.715570 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715538    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 19:06:50.715667 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.715645    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zdmr2\""
Sep 17 19:06:50.729965 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.729947    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrgzz"]
Sep 17 19:06:50.730054 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.730022    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.731520 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.731498    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4lr2c"]
Sep 17 19:06:50.732771 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.732753    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 19:06:50.732847 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.732753    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xbtpt\""
Sep 17 19:06:50.733005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.732982    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 19:06:50.733151 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.733132    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 19:06:50.733432 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.733414    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 19:06:50.794291 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.794262    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-djkgc"]
Sep 17 19:06:50.813416 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.813395    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:50.815449 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-installation-pull-secrets\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.815534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815459    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90fcbb05-90ab-44ec-b698-89859beb174f-ca-trust-extracted\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.815534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815478    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594853e-0435-4597-9471-416da5138d56-data-volume\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.815534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815510    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbc8v\" (UniqueName: \"kubernetes.io/projected/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-kube-api-access-zbc8v\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.815534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815534    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-bound-sa-token\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.815752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815644    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqcnj\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-kube-api-access-cqcnj\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.815752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815679    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6594853e-0435-4597-9471-416da5138d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.815752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815716    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-image-registry-private-configuration\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.815752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815739    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hgff\" (UniqueName: \"kubernetes.io/projected/6594853e-0435-4597-9471-416da5138d56-kube-api-access-4hgff\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815764    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-tmp-dir\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815785    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-trusted-ca\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815814    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-config-volume\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815832    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-metrics-tls\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815869    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6594853e-0435-4597-9471-416da5138d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-registry-tls\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815958    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-registry-certificates\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.816032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.815990    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6594853e-0435-4597-9471-416da5138d56-crio-socket\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.816412 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.816068    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/90fcbb05-90ab-44ec-b698-89859beb174f-ca-trust-extracted\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.816684 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.816667    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-5sk68\""
Sep 17 19:06:50.817461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.816689    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-config-volume\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.817461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.816917    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 19:06:50.817461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.817296    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-registry-certificates\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.817461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.817129    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 19:06:50.817461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.817384    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-tmp-dir\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.817777 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.817471    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/90fcbb05-90ab-44ec-b698-89859beb174f-trusted-ca\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.817777 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.817561    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 19:06:50.820109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.820084    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-metrics-tls\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.820212 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.820089    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-image-registry-private-configuration\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.820212 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.820148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/90fcbb05-90ab-44ec-b698-89859beb174f-installation-pull-secrets\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.820317 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.820293    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-registry-tls\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.829363 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.829342    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djkgc"]
Sep 17 19:06:50.837107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.837072    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbc8v\" (UniqueName: \"kubernetes.io/projected/a071f8b2-95a9-4f3d-a1f5-503b42fd3f59-kube-api-access-zbc8v\") pod \"dns-default-zrgzz\" (UID: \"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59\") " pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:50.837790 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.837772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqcnj\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-kube-api-access-cqcnj\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.838063 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.838040    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/90fcbb05-90ab-44ec-b698-89859beb174f-bound-sa-token\") pod \"image-registry-74c559dfd4-pcvpj\" (UID: \"90fcbb05-90ab-44ec-b698-89859beb174f\") " pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:50.917117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917059    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6594853e-0435-4597-9471-416da5138d56-crio-socket\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917088    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594853e-0435-4597-9471-416da5138d56-data-volume\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917109    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vtk4\" (UniqueName: \"kubernetes.io/projected/6c1b263c-5248-4467-b56a-f11354ad6519-kube-api-access-8vtk4\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:50.917265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917169    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6594853e-0435-4597-9471-416da5138d56-crio-socket\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917241    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6594853e-0435-4597-9471-416da5138d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917265    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hgff\" (UniqueName: \"kubernetes.io/projected/6594853e-0435-4597-9471-416da5138d56-kube-api-access-4hgff\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917285    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c1b263c-5248-4467-b56a-f11354ad6519-cert\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:50.917342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917320    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6594853e-0435-4597-9471-416da5138d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917354    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6594853e-0435-4597-9471-416da5138d56-data-volume\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.917805 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.917790    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6594853e-0435-4597-9471-416da5138d56-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.919562 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.919543    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6594853e-0435-4597-9471-416da5138d56-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.928341 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.928319    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hgff\" (UniqueName: \"kubernetes.io/projected/6594853e-0435-4597-9471-416da5138d56-kube-api-access-4hgff\") pod \"insights-runtime-extractor-4lr2c\" (UID: \"6594853e-0435-4597-9471-416da5138d56\") " pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:50.962902 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.962879    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:50.993999 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:50.993977    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:51.010279 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.010260    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:06:51.010279 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.010275    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:06:51.010398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.010362    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:06:51.014088 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014074    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:51.014175 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014124    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:06:51.014687 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014669    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:06:51.014795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014700    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jnl2p\""
Sep 17 19:06:51.014795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014782    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:06:51.014907 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014824    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:06:51.014954 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.014922    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjq9g\""
Sep 17 19:06:51.017821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.017802    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c1b263c-5248-4467-b56a-f11354ad6519-cert\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:51.017922 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.017856    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vtk4\" (UniqueName: \"kubernetes.io/projected/6c1b263c-5248-4467-b56a-f11354ad6519-kube-api-access-8vtk4\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:51.020084 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.020066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c1b263c-5248-4467-b56a-f11354ad6519-cert\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:51.021163 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.021143    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.027469 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.027454    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vtk4\" (UniqueName: \"kubernetes.io/projected/6c1b263c-5248-4467-b56a-f11354ad6519-kube-api-access-8vtk4\") pod \"ingress-canary-djkgc\" (UID: \"6c1b263c-5248-4467-b56a-f11354ad6519\") " pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:51.038407 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.038386    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4lr2c"
Sep 17 19:06:51.128860 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.128834    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-djkgc"
Sep 17 19:06:51.339395 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.339358    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zrgzz"]
Sep 17 19:06:51.341831 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.341813    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74c559dfd4-pcvpj"]
Sep 17 19:06:51.343870 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:51.343839    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda071f8b2_95a9_4f3d_a1f5_503b42fd3f59.slice/crio-bee477f4b1da732c24c7f1841e2e7ab7dcf6e82bb99091b94efc80df3864d489 WatchSource:0}: Error finding container bee477f4b1da732c24c7f1841e2e7ab7dcf6e82bb99091b94efc80df3864d489: Status 404 returned error can't find the container with id bee477f4b1da732c24c7f1841e2e7ab7dcf6e82bb99091b94efc80df3864d489
Sep 17 19:06:51.345569 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:51.345413    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90fcbb05_90ab_44ec_b698_89859beb174f.slice/crio-052d5a12464ca2884ca8354b8bc516ed798f06fe8d8db0893a063e912d3102f3 WatchSource:0}: Error finding container 052d5a12464ca2884ca8354b8bc516ed798f06fe8d8db0893a063e912d3102f3: Status 404 returned error can't find the container with id 052d5a12464ca2884ca8354b8bc516ed798f06fe8d8db0893a063e912d3102f3
Sep 17 19:06:51.348851 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.348825    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrgzz" event={"ID":"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59","Type":"ContainerStarted","Data":"bee477f4b1da732c24c7f1841e2e7ab7dcf6e82bb99091b94efc80df3864d489"}
Sep 17 19:06:51.349949 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.349928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj" event={"ID":"90fcbb05-90ab-44ec-b698-89859beb174f","Type":"ContainerStarted","Data":"052d5a12464ca2884ca8354b8bc516ed798f06fe8d8db0893a063e912d3102f3"}
Sep 17 19:06:51.365193 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.365172    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4lr2c"]
Sep 17 19:06:51.367982 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:51.367964    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6594853e_0435_4597_9471_416da5138d56.slice/crio-86d5eb6a326184b4527391e38154eb8516f3274c66241a1f7bac8e63f40e32d6 WatchSource:0}: Error finding container 86d5eb6a326184b4527391e38154eb8516f3274c66241a1f7bac8e63f40e32d6: Status 404 returned error can't find the container with id 86d5eb6a326184b4527391e38154eb8516f3274c66241a1f7bac8e63f40e32d6
Sep 17 19:06:51.421772 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:51.421746    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1b263c_5248_4467_b56a_f11354ad6519.slice/crio-950a20156172184d051300ee14d49c51c8431563d71b80bbc109d0ad6409327b WatchSource:0}: Error finding container 950a20156172184d051300ee14d49c51c8431563d71b80bbc109d0ad6409327b: Status 404 returned error can't find the container with id 950a20156172184d051300ee14d49c51c8431563d71b80bbc109d0ad6409327b
Sep 17 19:06:51.423389 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.423364    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-djkgc"]
Sep 17 19:06:51.485846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.485812    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.535205 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.535182    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:06:51.550974 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.550957    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.553393 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.553364    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:06:51.553787 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.553767    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 19:06:51.553899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.553795    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 19:06:51.554100 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.554082    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w7qv7\""
Sep 17 19:06:51.554199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.554123    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 19:06:51.554199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.554154    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 19:06:51.554357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.554339    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 19:06:51.554468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.554364    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 19:06:51.555090 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.555071    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 19:06:51.563703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.563683    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:51.621553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621531    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vmdz\" (UniqueName: \"kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.621890 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621590    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.621890 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621608    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.621890 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.621890 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621766    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.621890 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.621799    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722264    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722289    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vmdz\" (UniqueName: \"kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722545 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722336    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722545 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722353    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.722545 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.722370    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.723179 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.723157    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.723286 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.723157    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.723286 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.723272    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.724826 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.724804    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.724899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.724886    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.732384 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.732363    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vmdz\" (UniqueName: \"kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz\") pod \"console-7bcd9588cc-86b5r\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") " pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:51.860772 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:51.860739    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:06:52.062026 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.061971    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:52.126715 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.126521    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:52.172379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.172346    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:06:52.247082 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:52.247049    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05231422_7c54_437f_9391_bc779fc3d78f.slice/crio-9b9f7b131461978d04b8e18f5da773d545fb35597d5f032fa47fd0c6202b5099 WatchSource:0}: Error finding container 9b9f7b131461978d04b8e18f5da773d545fb35597d5f032fa47fd0c6202b5099: Status 404 returned error can't find the container with id 9b9f7b131461978d04b8e18f5da773d545fb35597d5f032fa47fd0c6202b5099
Sep 17 19:06:52.354751 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.353827    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djkgc" event={"ID":"6c1b263c-5248-4467-b56a-f11354ad6519","Type":"ContainerStarted","Data":"950a20156172184d051300ee14d49c51c8431563d71b80bbc109d0ad6409327b"}
Sep 17 19:06:52.356538 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.356513    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4lr2c" event={"ID":"6594853e-0435-4597-9471-416da5138d56","Type":"ContainerStarted","Data":"eb15a33c5dc02720f0b430281415f2744b5967b68d8ca1cf52f53ffe077e56cc"}
Sep 17 19:06:52.356671 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.356567    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4lr2c" event={"ID":"6594853e-0435-4597-9471-416da5138d56","Type":"ContainerStarted","Data":"86d5eb6a326184b4527391e38154eb8516f3274c66241a1f7bac8e63f40e32d6"}
Sep 17 19:06:52.358751 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.358727    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj" event={"ID":"90fcbb05-90ab-44ec-b698-89859beb174f","Type":"ContainerStarted","Data":"9f7d47a092ccc2a3f369df13f894c709c3cf3c9f3872bc1b5f4c7654a343f914"}
Sep 17 19:06:52.358868 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.358839    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:06:52.360794 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.360766    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd9588cc-86b5r" event={"ID":"05231422-7c54-437f-9391-bc779fc3d78f","Type":"ContainerStarted","Data":"9b9f7b131461978d04b8e18f5da773d545fb35597d5f032fa47fd0c6202b5099"}
Sep 17 19:06:52.383313 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:52.383271    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj" podStartSLOduration=3.383256846 podStartE2EDuration="3.383256846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:52.382167965 +0000 UTC m=+57.547909241" watchObservedRunningTime="2026-09-17 19:06:52.383256846 +0000 UTC m=+57.548998120"
Sep 17 19:06:53.022008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.021971    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.243339 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.243270    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.251231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.250192    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"]
Sep 17 19:06:53.253113 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.252383    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:53.255822 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.255616    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-gv6xt\""
Sep 17 19:06:53.256224 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.256205    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 17 19:06:53.264904 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.264853    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"]
Sep 17 19:06:53.343276 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.342469    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qksgz\" (UID: \"f85a0321-e71f-4edf-af37-18cffea92d4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:53.373479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.373444    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4lr2c" event={"ID":"6594853e-0435-4597-9471-416da5138d56","Type":"ContainerStarted","Data":"639f8378fcf85e30ddc2dd03897432d6d822b7ce50ad0c3e0860ff2f004b3bc1"}
Sep 17 19:06:53.378309 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.378273    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrgzz" event={"ID":"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59","Type":"ContainerStarted","Data":"de9217e0ceea1e05abbe0c4e65123c5da0a26c310ab7da0f079c88d345ac208a"}
Sep 17 19:06:53.378434 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.378323    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zrgzz" event={"ID":"a071f8b2-95a9-4f3d-a1f5-503b42fd3f59","Type":"ContainerStarted","Data":"42074c8019b46732c031e20df9d7514f32611c0f1c34c11ec02f576a8f9a1922"}
Sep 17 19:06:53.446942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.444048    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qksgz\" (UID: \"f85a0321-e71f-4edf-af37-18cffea92d4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:53.446942 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:53.444187    2586 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 19:06:53.446942 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:06:53.444246    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates podName:f85a0321-e71f-4edf-af37-18cffea92d4b nodeName:}" failed. No retries permitted until 2026-09-17 19:06:53.944227516 +0000 UTC m=+59.109968775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates") pod "prometheus-operator-admission-webhook-5c6747dc7b-qksgz" (UID: "f85a0321-e71f-4edf-af37-18cffea92d4b") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 17 19:06:53.534068 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.533976    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.579008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.576837    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zrgzz" podStartSLOduration=1.840816089 podStartE2EDuration="3.576813794s" totalImagesPullingTime="1.735997705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:53.420146666 +0000 UTC m=+58.585887958" watchObservedRunningTime="2026-09-17 19:06:53.576813794 +0000 UTC m=+58.742555066"
Sep 17 19:06:53.579008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.577942    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:06:53.582784 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.580769    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.603794 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.603770    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 19:06:53.604765 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.604742    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.645761    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.645813    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.645912    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.645947    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb8pt\" (UniqueName: \"kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.645978    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.646008    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.646044    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.646951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.646921    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.722378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.722344    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746492    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746534    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb8pt\" (UniqueName: \"kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746561    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746591    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746642    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746673    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.746707    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.748109 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.747474    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.750765 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.749364    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.750765 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.750167    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.750765 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.750719    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.754996 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.754952    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.755477 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.755455    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.768816 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.768775    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb8pt\" (UniqueName: \"kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt\") pod \"console-65496b889c-p2rdc\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") " pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.812717 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.812644    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.880365 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.880325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:53.896165 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.896140    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:06:53.948713 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.948676    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qksgz\" (UID: \"f85a0321-e71f-4edf-af37-18cffea92d4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:53.952941 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:53.952914    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f85a0321-e71f-4edf-af37-18cffea92d4b-tls-certificates\") pod \"prometheus-operator-admission-webhook-5c6747dc7b-qksgz\" (UID: \"f85a0321-e71f-4edf-af37-18cffea92d4b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:54.185040 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:54.184730    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:54.382092 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:54.382064    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:06:55.295123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:55.295081    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:55.359254 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:55.359217    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:56.295989 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:56.295959    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6p2vh"
Sep 17 19:06:56.494575 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:56.494546    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:56.602854 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:56.602777    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:56.658312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:56.658267    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:57.487082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:57.487025    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:57.553902 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:57.553873    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:06:57.832997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:57.832967    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"]
Sep 17 19:06:57.835881 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:57.835852    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf85a0321_e71f_4edf_af37_18cffea92d4b.slice/crio-c93d62821a8b2b91ad9212cfab24935339a7af72cd37ea62d76b64df2e2b6564 WatchSource:0}: Error finding container c93d62821a8b2b91ad9212cfab24935339a7af72cd37ea62d76b64df2e2b6564: Status 404 returned error can't find the container with id c93d62821a8b2b91ad9212cfab24935339a7af72cd37ea62d76b64df2e2b6564
Sep 17 19:06:57.852488 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:57.852464    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:06:57.856111 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:06:57.856091    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e99b5c4_d53c_480d_83cc_b2196d895602.slice/crio-e428305190540d2e9507627a42b2bcb62ef75294af97a6fd440f893e74e3ed6b WatchSource:0}: Error finding container e428305190540d2e9507627a42b2bcb62ef75294af97a6fd440f893e74e3ed6b: Status 404 returned error can't find the container with id e428305190540d2e9507627a42b2bcb62ef75294af97a6fd440f893e74e3ed6b
Sep 17 19:06:58.397046 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.397009    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-djkgc" event={"ID":"6c1b263c-5248-4467-b56a-f11354ad6519","Type":"ContainerStarted","Data":"347921cf7ef82dd4cfb668079476eb53909aa34fa3878ff45071b757e40845dd"}
Sep 17 19:06:58.399205 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.399182    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4lr2c" event={"ID":"6594853e-0435-4597-9471-416da5138d56","Type":"ContainerStarted","Data":"446e570bb9f2d3e08ee84210236898fc0f99a0b5837891bf324828c728446b5d"}
Sep 17 19:06:58.400370 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.400350    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz" event={"ID":"f85a0321-e71f-4edf-af37-18cffea92d4b","Type":"ContainerStarted","Data":"c93d62821a8b2b91ad9212cfab24935339a7af72cd37ea62d76b64df2e2b6564"}
Sep 17 19:06:58.401834 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.401814    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd9588cc-86b5r" event={"ID":"05231422-7c54-437f-9391-bc779fc3d78f","Type":"ContainerStarted","Data":"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"}
Sep 17 19:06:58.403211 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.403191    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65496b889c-p2rdc" event={"ID":"6e99b5c4-d53c-480d-83cc-b2196d895602","Type":"ContainerStarted","Data":"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"}
Sep 17 19:06:58.403286 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.403224    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65496b889c-p2rdc" event={"ID":"6e99b5c4-d53c-480d-83cc-b2196d895602","Type":"ContainerStarted","Data":"e428305190540d2e9507627a42b2bcb62ef75294af97a6fd440f893e74e3ed6b"}
Sep 17 19:06:58.445718 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.445680    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-djkgc" podStartSLOduration=4.014916703 podStartE2EDuration="8.445666017s" totalImagesPullingTime="4.430749314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.422560904 +0000 UTC m=+63.588302197" watchObservedRunningTime="2026-09-17 19:06:58.445666017 +0000 UTC m=+63.611407291"
Sep 17 19:06:58.472153 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.472087    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4lr2c" podStartSLOduration=2.468564014 podStartE2EDuration="8.47207336s" totalImagesPullingTime="6.003509346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:06:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.470927408 +0000 UTC m=+63.636668701" watchObservedRunningTime="2026-09-17 19:06:58.47207336 +0000 UTC m=+63.637814635"
Sep 17 19:06:58.472379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:58.472357    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-65496b889c-p2rdc" podStartSLOduration=5.472349117 podStartE2EDuration="5.472349117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.446119856 +0000 UTC m=+63.611861151" watchObservedRunningTime="2026-09-17 19:06:58.472349117 +0000 UTC m=+63.638090391"
Sep 17 19:06:59.408670 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:59.408610    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz" event={"ID":"f85a0321-e71f-4edf-af37-18cffea92d4b","Type":"ContainerStarted","Data":"8bd4e50b8b144704ed3213e367c45093b586c6b401af5da2d6f124d97e132c5b"}
Sep 17 19:06:59.409191 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:59.409172    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:59.413422 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:59.413389    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz"
Sep 17 19:06:59.431821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:59.431747    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bcd9588cc-86b5r" podStartSLOduration=3.070942373 podStartE2EDuration="8.431732967s" totalImagesPullingTime="5.360790594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:58.499162859 +0000 UTC m=+63.664904134" watchObservedRunningTime="2026-09-17 19:06:59.431732967 +0000 UTC m=+64.597474242"
Sep 17 19:06:59.464688 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:06:59.464647    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-5c6747dc7b-qksgz" podStartSLOduration=5.125133523 podStartE2EDuration="6.464606344s" totalImagesPullingTime="1.339472821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:06:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:06:59.436060445 +0000 UTC m=+64.601801719" watchObservedRunningTime="2026-09-17 19:06:59.464606344 +0000 UTC m=+64.630347619"
Sep 17 19:07:00.265388 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.265351    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.330305 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.330279    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.467602 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.467577    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.572951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.572927    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.669088 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.669071    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.772636 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.772601    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.867741 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.867679    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:00.971658 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:00.971613    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:01.861119 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:01.861081    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:07:01.861119 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:01.861138    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:07:01.862431 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:01.862404    2586 patch_prober.go:28] interesting pod/console-7bcd9588cc-86b5r container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused" start-of-body=
Sep 17 19:07:01.862505 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:01.862451    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bcd9588cc-86b5r" podUID="05231422-7c54-437f-9391-bc779fc3d78f" containerName="console" probeResult="failure" output="Get \"https://10.133.0.10:8443/health\": dial tcp 10.133.0.10:8443: connect: connection refused"
Sep 17 19:07:02.815924 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.815892    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:07:02.815924 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.815928    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:07:02.819234 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.819216    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 19:07:02.819987 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.819974    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 19:07:02.828381 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.828355    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dd1f444b-0e61-471b-9a77-6db913e39d7d-original-pull-secret\") pod \"global-pull-secret-syncer-vnlqc\" (UID: \"dd1f444b-0e61-471b-9a77-6db913e39d7d\") " pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:07:02.828464 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.828384    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b563966d-464c-4962-9c8e-71ba17af1708-metrics-certs\") pod \"network-metrics-daemon-m2gkt\" (UID: \"b563966d-464c-4962-9c8e-71ba17af1708\") " pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:07:02.916848 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.916821    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:07:02.919882 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.919866    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 19:07:02.929420 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.929405    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 19:07:02.939978 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:02.939954    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mr9t\" (UniqueName: \"kubernetes.io/projected/3d798031-06ea-48a0-90e7-f4c6928307d1-kube-api-access-8mr9t\") pod \"network-check-target-m86xm\" (UID: \"3d798031-06ea-48a0-90e7-f4c6928307d1\") " pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:07:03.020069 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.020050    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnlqc"
Sep 17 19:07:03.028568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.028552    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sjq9g\""
Sep 17 19:07:03.033756 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.033740    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-jnl2p\""
Sep 17 19:07:03.036870 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.036856    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-m2gkt"
Sep 17 19:07:03.042041 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.042030    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:07:03.226592 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.226564    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:03.322908 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.322884    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnlqc"]
Sep 17 19:07:03.325804 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:03.325780    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd1f444b_0e61_471b_9a77_6db913e39d7d.slice/crio-b41b3eb2c78b6cb778349df87a537dbaee923426bccb74a068e0d7b0ecb949fb WatchSource:0}: Error finding container b41b3eb2c78b6cb778349df87a537dbaee923426bccb74a068e0d7b0ecb949fb: Status 404 returned error can't find the container with id b41b3eb2c78b6cb778349df87a537dbaee923426bccb74a068e0d7b0ecb949fb
Sep 17 19:07:03.357152 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.357129    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-m2gkt"]
Sep 17 19:07:03.359785 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:03.359759    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb563966d_464c_4962_9c8e_71ba17af1708.slice/crio-0aee883862f2a7d55fe207dbc125cd7187e3b8ff27528c2b1748b537404b98cf WatchSource:0}: Error finding container 0aee883862f2a7d55fe207dbc125cd7187e3b8ff27528c2b1748b537404b98cf: Status 404 returned error can't find the container with id 0aee883862f2a7d55fe207dbc125cd7187e3b8ff27528c2b1748b537404b98cf
Sep 17 19:07:03.400427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.400392    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-m86xm"]
Sep 17 19:07:03.402513 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:03.402495    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d798031_06ea_48a0_90e7_f4c6928307d1.slice/crio-c805155d4c212f3e857f6d0f1a6173693dd4f5394e728e400161acb5ab34dba3 WatchSource:0}: Error finding container c805155d4c212f3e857f6d0f1a6173693dd4f5394e728e400161acb5ab34dba3: Status 404 returned error can't find the container with id c805155d4c212f3e857f6d0f1a6173693dd4f5394e728e400161acb5ab34dba3
Sep 17 19:07:03.421475 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.421451    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m86xm" event={"ID":"3d798031-06ea-48a0-90e7-f4c6928307d1","Type":"ContainerStarted","Data":"c805155d4c212f3e857f6d0f1a6173693dd4f5394e728e400161acb5ab34dba3"}
Sep 17 19:07:03.422595 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.422571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2gkt" event={"ID":"b563966d-464c-4962-9c8e-71ba17af1708","Type":"ContainerStarted","Data":"0aee883862f2a7d55fe207dbc125cd7187e3b8ff27528c2b1748b537404b98cf"}
Sep 17 19:07:03.423598 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.423577    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnlqc" event={"ID":"dd1f444b-0e61-471b-9a77-6db913e39d7d","Type":"ContainerStarted","Data":"b41b3eb2c78b6cb778349df87a537dbaee923426bccb74a068e0d7b0ecb949fb"}
Sep 17 19:07:03.897659 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.897000    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:03.897659 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.897074    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:03.898426 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.898398    2586 patch_prober.go:28] interesting pod/console-65496b889c-p2rdc container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.133.0.12:8443/health\": dial tcp 10.133.0.12:8443: connect: connection refused" start-of-body=
Sep 17 19:07:03.898546 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:03.898453    2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-65496b889c-p2rdc" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerName="console" probeResult="failure" output="Get \"https://10.133.0.12:8443/health\": dial tcp 10.133.0.12:8443: connect: connection refused"
Sep 17 19:07:04.388304 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:04.388275    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zrgzz"
Sep 17 19:07:05.436513 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.436472    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2gkt" event={"ID":"b563966d-464c-4962-9c8e-71ba17af1708","Type":"ContainerStarted","Data":"809934ab300a7d4c87e3ce304b84f7602ca4b761098a531c0122da01b2b6c3b3"}
Sep 17 19:07:05.437012 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.436535    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-m2gkt" event={"ID":"b563966d-464c-4962-9c8e-71ba17af1708","Type":"ContainerStarted","Data":"b1d57df31109f38d1965f29832dc4d0e3d139410f5a2b1741da27818a8ebfb50"}
Sep 17 19:07:05.462201 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.462151    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-m2gkt" podStartSLOduration=67.227829071 podStartE2EDuration="1m8.462137137s" totalImagesPullingTime="1.234308066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:05.460098283 +0000 UTC m=+70.625839587" watchObservedRunningTime="2026-09-17 19:07:05.462137137 +0000 UTC m=+70.627878412"
Sep 17 19:07:05.474766 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.474737    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:05.690061 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.689920    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:05.776057 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.776022    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:05.795000 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.794962    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6tthb"]
Sep 17 19:07:05.797962 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.797936    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.800266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.800241    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 19:07:05.800957 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.800926    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 19:07:05.801237 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.801218    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6bgp8\""
Sep 17 19:07:05.801336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.801240    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 19:07:05.801336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.801264    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 19:07:05.801529 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.801496    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 19:07:05.801529 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.801525    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 19:07:05.838289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838244    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctdb\" (UniqueName: \"kubernetes.io/projected/332b25c0-1317-475c-8bc2-e5dacfee288b-kube-api-access-tctdb\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838286    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-root\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838305    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-textfile\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838326    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-sys\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838352    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-tls\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838381    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-wtmp\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838405    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-metrics-client-ca\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838429    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.838486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.838445    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.876778 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.876745    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:05.939073 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939039    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-wtmp\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939083    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-metrics-client-ca\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939108    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939165    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctdb\" (UniqueName: \"kubernetes.io/projected/332b25c0-1317-475c-8bc2-e5dacfee288b-kube-api-access-tctdb\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939190    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-root\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939212    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-textfile\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939235    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-sys\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939269    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-tls\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939771    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-root\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.939938 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939921    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-wtmp\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.940001 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.939980    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/332b25c0-1317-475c-8bc2-e5dacfee288b-sys\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.940330 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.940104    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-textfile\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.940459 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.940437    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-accelerators-collector-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.940528 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.940453    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/332b25c0-1317-475c-8bc2-e5dacfee288b-metrics-client-ca\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.942248 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.942220    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-tls\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.942337 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.942305    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/332b25c0-1317-475c-8bc2-e5dacfee288b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:05.950757 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:05.949584    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctdb\" (UniqueName: \"kubernetes.io/projected/332b25c0-1317-475c-8bc2-e5dacfee288b-kube-api-access-tctdb\") pod \"node-exporter-6tthb\" (UID: \"332b25c0-1317-475c-8bc2-e5dacfee288b\") " pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:06.001183 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.001149    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.110249 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.110211    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6tthb"
Sep 17 19:07:06.145465 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.145436    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.202988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.202909    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.520979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.520903    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.729523 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.729490    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:06.826806 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:06.826770    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:07.073987 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:07.073949    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:07.175652 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:07.175561    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:07.330728 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:07.330696    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:08.125831 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:08.125745    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332b25c0_1317_475c_8bc2_e5dacfee288b.slice/crio-aec4a537d07e93fbdfe5e0d7647fea14b26c828f1d5d0b10434bf77301213f1e WatchSource:0}: Error finding container aec4a537d07e93fbdfe5e0d7647fea14b26c828f1d5d0b10434bf77301213f1e: Status 404 returned error can't find the container with id aec4a537d07e93fbdfe5e0d7647fea14b26c828f1d5d0b10434bf77301213f1e
Sep 17 19:07:08.451392 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.451287    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnlqc" event={"ID":"dd1f444b-0e61-471b-9a77-6db913e39d7d","Type":"ContainerStarted","Data":"7bca31123ed2ad803b7b737e4474fc46119f900592f16a8937100793bd83dd0c"}
Sep 17 19:07:08.452964 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.452938    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-m86xm" event={"ID":"3d798031-06ea-48a0-90e7-f4c6928307d1","Type":"ContainerStarted","Data":"e41ce830b5a85778fc6c93b8b38393ce8fd0fb5bd2044dba7431338055b722f6"}
Sep 17 19:07:08.453100 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.453022    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:07:08.454169 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.454150    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tthb" event={"ID":"332b25c0-1317-475c-8bc2-e5dacfee288b","Type":"ContainerStarted","Data":"aec4a537d07e93fbdfe5e0d7647fea14b26c828f1d5d0b10434bf77301213f1e"}
Sep 17 19:07:08.471135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.471085    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vnlqc" podStartSLOduration=66.661470151 podStartE2EDuration="1m11.47107305s" totalImagesPullingTime="4.809602899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:08.470323131 +0000 UTC m=+73.636064407" watchObservedRunningTime="2026-09-17 19:07:08.47107305 +0000 UTC m=+73.636814321"
Sep 17 19:07:08.922714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:08.922682    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.032937 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.032909    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.233377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.233312    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.451371 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.451342    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.459922 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.459897    2586 generic.go:356] "Generic (PLEG): container finished" podID="332b25c0-1317-475c-8bc2-e5dacfee288b" containerID="c70bbea5065a5c0573eddb7864b7ce8b236ecd27817171843b10d64021ecbc5e" exitCode=0
Sep 17 19:07:09.460048 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.459972    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tthb" event={"ID":"332b25c0-1317-475c-8bc2-e5dacfee288b","Type":"ContainerDied","Data":"c70bbea5065a5c0573eddb7864b7ce8b236ecd27817171843b10d64021ecbc5e"}
Sep 17 19:07:09.490035 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.489953    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-m86xm" podStartSLOduration=67.766481919 podStartE2EDuration="1m12.489935471s" totalImagesPullingTime="4.723453552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:08.502577519 +0000 UTC m=+73.668318806" watchObservedRunningTime="2026-09-17 19:07:09.489935471 +0000 UTC m=+74.655676748"
Sep 17 19:07:09.593808 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.593775    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:09.679565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:09.679524    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.109535 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.109505    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8577cd5c85-m5rqn"]
Sep 17 19:07:10.112512 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.112498    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.118813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.118793    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 17 19:07:10.118993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.118976    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 17 19:07:10.119089 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.119074    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 17 19:07:10.120001 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.119981    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mpcz8\""
Sep 17 19:07:10.120105 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.119995    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 17 19:07:10.120105 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.120070    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bq2bjmealaih2\""
Sep 17 19:07:10.171497 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-client-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171591 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171591 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171542    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-audit-log\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171594    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-tls\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171641    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-metrics-server-audit-profiles\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171702    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-client-certs\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.171749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.171734    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgs9s\" (UniqueName: \"kubernetes.io/projected/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-kube-api-access-cgs9s\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.204537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.204516    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8577cd5c85-m5rqn"]
Sep 17 19:07:10.272883 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.272860    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-tls\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.272889    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-metrics-server-audit-profiles\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.272918    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-client-certs\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.272947    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgs9s\" (UniqueName: \"kubernetes.io/projected/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-kube-api-access-cgs9s\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.273076    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-client-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.273120    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.273157    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-audit-log\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273546 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.273502    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-audit-log\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.273860 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.273837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.274251 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.274228    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-metrics-server-audit-profiles\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.275527 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.275508    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-tls\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.275783 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.275764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-secret-metrics-server-client-certs\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.275829 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.275772    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-client-ca-bundle\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.291820 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.291803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgs9s\" (UniqueName: \"kubernetes.io/projected/dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac-kube-api-access-cgs9s\") pod \"metrics-server-8577cd5c85-m5rqn\" (UID: \"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac\") " pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.420765 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.420713    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:10.466326 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.466291    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tthb" event={"ID":"332b25c0-1317-475c-8bc2-e5dacfee288b","Type":"ContainerStarted","Data":"b327440e65e3683531d36658605b205920264d102ad61072c891f8b04d337646"}
Sep 17 19:07:10.466479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.466349    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6tthb" event={"ID":"332b25c0-1317-475c-8bc2-e5dacfee288b","Type":"ContainerStarted","Data":"b801b137b69ef7d7f66b6b8043cd5dd7e612ed5160ffe07ade097a7b2b109c28"}
Sep 17 19:07:10.506074 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.506015    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6tthb" podStartSLOduration=4.63902219 podStartE2EDuration="5.505992934s" totalImagesPullingTime="798.524402ms" totalInitContainerRuntime="68.446342ms" isStatefulPod=true podCreationTimestamp="2026-09-17 19:07:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:10.505332474 +0000 UTC m=+75.671073768" watchObservedRunningTime="2026-09-17 19:07:10.505992934 +0000 UTC m=+75.671734211"
Sep 17 19:07:10.567778 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.567751    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.640829 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.640801    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:07:10.675483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.675421    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:07:10.678947 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.678919    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.679528 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.679514    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.724327 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.724305    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:07:10.725323 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.725299    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8577cd5c85-m5rqn"]
Sep 17 19:07:10.725990 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:10.725970    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0ce4fe_2e74_45ba_893e_5bbcd7e567ac.slice/crio-afbcfa80a48c20c252355bd6cab7a1603d24ff98395a99fd3930e162c5f5588d WatchSource:0}: Error finding container afbcfa80a48c20c252355bd6cab7a1603d24ff98395a99fd3930e162c5f5588d: Status 404 returned error can't find the container with id afbcfa80a48c20c252355bd6cab7a1603d24ff98395a99fd3930e162c5f5588d
Sep 17 19:07:10.777740 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m22dl\" (UniqueName: \"kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777773    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777900 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777826    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777900 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777870    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777910    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.777959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.777949    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.810121 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.810099    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:10.878483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878461    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878489    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878516    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878552    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878727 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878667    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878782 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878726    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.878830 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.878781    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m22dl\" (UniqueName: \"kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.879301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.879278    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.879405 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.879301    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.879405 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.879275    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.879564 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.879545    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.881731 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.881715    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.881827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.881811    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.888401 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.888380    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m22dl\" (UniqueName: \"kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl\") pod \"console-6cd77f78-2clvc\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") " pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:10.988231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:10.988167    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:11.003192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.003168    2586 patch_prober.go:28] interesting pod/image-registry-74c559dfd4-pcvpj container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 19:07:11.003302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.003212    2586 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj" podUID="90fcbb05-90ab-44ec-b698-89859beb174f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 19:07:11.259360 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.259286    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:07:11.262144 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:07:11.262118    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe14cc17_3d41_4f62_be82_27df48afb593.slice/crio-2f2d9f4f363c7d74d6dd305e4f7c49a41ba22409faeb57352ec74ff79b4305ac WatchSource:0}: Error finding container 2f2d9f4f363c7d74d6dd305e4f7c49a41ba22409faeb57352ec74ff79b4305ac: Status 404 returned error can't find the container with id 2f2d9f4f363c7d74d6dd305e4f7c49a41ba22409faeb57352ec74ff79b4305ac
Sep 17 19:07:11.471469 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.471437    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn" event={"ID":"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac","Type":"ContainerStarted","Data":"afbcfa80a48c20c252355bd6cab7a1603d24ff98395a99fd3930e162c5f5588d"}
Sep 17 19:07:11.472945 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.472908    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd77f78-2clvc" event={"ID":"fe14cc17-3d41-4f62-be82-27df48afb593","Type":"ContainerStarted","Data":"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"}
Sep 17 19:07:11.473052 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.472965    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd77f78-2clvc" event={"ID":"fe14cc17-3d41-4f62-be82-27df48afb593","Type":"ContainerStarted","Data":"2f2d9f4f363c7d74d6dd305e4f7c49a41ba22409faeb57352ec74ff79b4305ac"}
Sep 17 19:07:11.493488 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:11.493441    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6cd77f78-2clvc" podStartSLOduration=1.493426391 podStartE2EDuration="1.493426391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:11.492340257 +0000 UTC m=+76.658081559" watchObservedRunningTime="2026-09-17 19:07:11.493426391 +0000 UTC m=+76.659167662"
Sep 17 19:07:12.219451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:12.219406    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:12.591850 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:12.591822    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:12.653776 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:12.653752    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:13.382456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.382428    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74c559dfd4-pcvpj"
Sep 17 19:07:13.484352 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.484324    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn" event={"ID":"dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac","Type":"ContainerStarted","Data":"d15f2d04e6c160e6fe579256b89102dd1e3ddbed65825a1e7b829bcf2fb9f339"}
Sep 17 19:07:13.512298 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.512252    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn" podStartSLOduration=1.8214319350000001 podStartE2EDuration="3.512237182s" totalImagesPullingTime="1.690805247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:07:13.509804167 +0000 UTC m=+78.675545467" watchObservedRunningTime="2026-09-17 19:07:13.512237182 +0000 UTC m=+78.677978504"
Sep 17 19:07:13.560503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.560466    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:13.619560 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.619535    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:13.717131 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.717070    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:13.902201 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.902169    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:13.905947 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:13.905926    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:14.073188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:14.073157    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:14.135791 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:14.135768    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.395083 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:16.395045    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:16.469089 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:16.469058    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:17.701144 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:17.701114    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:17.775083 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:17.775054    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.580770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:18.580745    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.791297 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:18.791265    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.887396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:18.887330    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:18.986316 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:18.986286    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:19.261216 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:19.261143    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:19.327761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:19.327724    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:20.988871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:20.988836    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:20.989235 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:20.988901    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:20.993718 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:20.993695    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:21.495979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.495943    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:21.515153 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.515127    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:07:21.582382 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.582353    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:07:21.588962 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.588938    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:21.740235 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.740204    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:21.843824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.843797    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:21.936128 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:21.936099    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.034985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:22.034955    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.141813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:22.141743    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.453423 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:22.453350    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.553766 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:22.553737    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:22.655517 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:22.655490    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:25.503431 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:25.503400    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:25.562452 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:25.562404    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.421336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:30.421299    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:30.421793 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:30.421361    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:30.645000 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:30.644953    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.715238 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:30.715164    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:30.784877 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:30.784856    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:35.659433 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:35.659390    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bcd9588cc-86b5r" podUID="05231422-7c54-437f-9391-bc779fc3d78f" containerName="console" containerID="cri-o://5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15" gracePeriod=15
Sep 17 19:07:35.900392 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:35.900370    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:07:36.058946 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.058917    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.058950    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.058973    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059018    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vmdz\" (UniqueName: \"kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059052    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059070    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config\") pod \"05231422-7c54-437f-9391-bc779fc3d78f\" (UID: \"05231422-7c54-437f-9391-bc779fc3d78f\") "
Sep 17 19:07:36.059504 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059474    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:36.059654 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059579    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca" (OuterVolumeSpecName: "service-ca") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:36.059654 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.059593    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config" (OuterVolumeSpecName: "console-config") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:36.061493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.061472    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:36.061586 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.061479    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz" (OuterVolumeSpecName: "kube-api-access-5vmdz") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "kube-api-access-5vmdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:36.061586 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.061551    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "05231422-7c54-437f-9391-bc779fc3d78f" (UID: "05231422-7c54-437f-9391-bc779fc3d78f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:36.159929 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159899    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-service-ca\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.159929 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159926    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-console-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.160076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159935    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/05231422-7c54-437f-9391-bc779fc3d78f-oauth-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.160076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159945    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-oauth-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.160076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159953    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/05231422-7c54-437f-9391-bc779fc3d78f-console-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.160076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.159961    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vmdz\" (UniqueName: \"kubernetes.io/projected/05231422-7c54-437f-9391-bc779fc3d78f-kube-api-access-5vmdz\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:36.561958 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.561928    2586 generic.go:356] "Generic (PLEG): container finished" podID="05231422-7c54-437f-9391-bc779fc3d78f" containerID="5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15" exitCode=0
Sep 17 19:07:36.562145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.561981    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcd9588cc-86b5r"
Sep 17 19:07:36.562145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.561998    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd9588cc-86b5r" event={"ID":"05231422-7c54-437f-9391-bc779fc3d78f","Type":"ContainerDied","Data":"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"}
Sep 17 19:07:36.562145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.562048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcd9588cc-86b5r" event={"ID":"05231422-7c54-437f-9391-bc779fc3d78f","Type":"ContainerDied","Data":"9b9f7b131461978d04b8e18f5da773d545fb35597d5f032fa47fd0c6202b5099"}
Sep 17 19:07:36.562145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.562087    2586 scope.go:117] "RemoveContainer" containerID="5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"
Sep 17 19:07:36.571188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.571169    2586 scope.go:117] "RemoveContainer" containerID="5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"
Sep 17 19:07:36.571487 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:07:36.571461    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15\": container with ID starting with 5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15 not found: ID does not exist" containerID="5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"
Sep 17 19:07:36.571571 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.571494    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15"} err="failed to get container status \"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15\": rpc error: code = NotFound desc = could not find container \"5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15\": container with ID starting with 5a3aac0b884b06cc91c6667ea15723501faa9d7205eae4bb1b06d5dfb745db15 not found: ID does not exist"
Sep 17 19:07:36.585905 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.585860    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:07:36.587084 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:36.587056    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcd9588cc-86b5r"]
Sep 17 19:07:37.013848 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:37.013772    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05231422-7c54-437f-9391-bc779fc3d78f" path="/var/lib/kubelet/pods/05231422-7c54-437f-9391-bc779fc3d78f/volumes"
Sep 17 19:07:37.054559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:37.054530    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:37.111974 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:37.111950    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:39.462637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:39.462588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-m86xm"
Sep 17 19:07:46.601976 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.601936    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-65496b889c-p2rdc" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerName="console" containerID="cri-o://dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e" gracePeriod=15
Sep 17 19:07:46.843769 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.843745    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:46.930250 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930185    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lb8pt\" (UniqueName: \"kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930250 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930232    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930442 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930252    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930442 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930271    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930442 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930288    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930594 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930455    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930594 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930533    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca\") pod \"6e99b5c4-d53c-480d-83cc-b2196d895602\" (UID: \"6e99b5c4-d53c-480d-83cc-b2196d895602\") "
Sep 17 19:07:46.930714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930695    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config" (OuterVolumeSpecName: "console-config") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:46.930768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930705    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:46.930768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930727    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:46.930877 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930838    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-trusted-ca-bundle\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:46.930877 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.930857    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-oauth-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:46.931104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.931079    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca" (OuterVolumeSpecName: "service-ca") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:07:46.931389 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.931130    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-console-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:46.932597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.932568    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:46.932743 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.932599    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt" (OuterVolumeSpecName: "kube-api-access-lb8pt") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "kube-api-access-lb8pt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:07:46.932743 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:46.932692    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6e99b5c4-d53c-480d-83cc-b2196d895602" (UID: "6e99b5c4-d53c-480d-83cc-b2196d895602"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:07:47.032297 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.032271    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6e99b5c4-d53c-480d-83cc-b2196d895602-service-ca\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:47.032297 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.032294    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb8pt\" (UniqueName: \"kubernetes.io/projected/6e99b5c4-d53c-480d-83cc-b2196d895602-kube-api-access-lb8pt\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:47.032435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.032303    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-oauth-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:47.032435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.032311    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6e99b5c4-d53c-480d-83cc-b2196d895602-console-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:07:47.600781 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.600752    2586 generic.go:356] "Generic (PLEG): container finished" podID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerID="dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e" exitCode=0
Sep 17 19:07:47.600944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.600796    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-65496b889c-p2rdc"
Sep 17 19:07:47.600944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.600806    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65496b889c-p2rdc" event={"ID":"6e99b5c4-d53c-480d-83cc-b2196d895602","Type":"ContainerDied","Data":"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"}
Sep 17 19:07:47.600944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.600845    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-65496b889c-p2rdc" event={"ID":"6e99b5c4-d53c-480d-83cc-b2196d895602","Type":"ContainerDied","Data":"e428305190540d2e9507627a42b2bcb62ef75294af97a6fd440f893e74e3ed6b"}
Sep 17 19:07:47.600944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.600862    2586 scope.go:117] "RemoveContainer" containerID="dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"
Sep 17 19:07:47.608853 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.608766    2586 scope.go:117] "RemoveContainer" containerID="dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"
Sep 17 19:07:47.609073 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:07:47.609022    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e\": container with ID starting with dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e not found: ID does not exist" containerID="dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"
Sep 17 19:07:47.609073 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.609048    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e"} err="failed to get container status \"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e\": rpc error: code = NotFound desc = could not find container \"dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e\": container with ID starting with dd5e96e26d1293c353ba5ea32778a4219c7ce61b080a5eaeaf806b4e1f2d697e not found: ID does not exist"
Sep 17 19:07:47.619128 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.619109    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:07:47.627405 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:47.627380    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-65496b889c-p2rdc"]
Sep 17 19:07:49.013927 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:49.013894    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" path="/var/lib/kubelet/pods/6e99b5c4-d53c-480d-83cc-b2196d895602/volumes"
Sep 17 19:07:50.426050 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:50.426021    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:50.429666 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:50.429648    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8577cd5c85-m5rqn"
Sep 17 19:07:55.315082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:55.315047    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.551127 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:55.551098    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.752268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:55.752182    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:07:55.811247 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:07:55.811213    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:00.447830 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.447801    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-655f6986b-dbx6l"]
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448025    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerName="console"
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448036    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerName="console"
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448045    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05231422-7c54-437f-9391-bc779fc3d78f" containerName="console"
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448050    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="05231422-7c54-437f-9391-bc779fc3d78f" containerName="console"
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448083    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="05231422-7c54-437f-9391-bc779fc3d78f" containerName="console"
Sep 17 19:08:00.448192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.448090    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e99b5c4-d53c-480d-83cc-b2196d895602" containerName="console"
Sep 17 19:08:00.451161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.451144    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.454933 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.454910    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 17 19:08:00.455088 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.455072    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-zg6mx\""
Sep 17 19:08:00.455578 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.455566    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 17 19:08:00.456303 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.456285    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 17 19:08:00.456404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.456362    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 17 19:08:00.456795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.456776    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 17 19:08:00.461314 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.461295    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 17 19:08:00.477800 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.477771    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655f6986b-dbx6l"]
Sep 17 19:08:00.623486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623457    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-metrics-client-ca\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623487    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzf8q\" (UniqueName: \"kubernetes.io/projected/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-kube-api-access-xzf8q\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623710 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623506    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623710 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623651    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-serving-certs-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623710 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623689    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623847 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623718    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623847 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623746    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.623847 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.623777    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-federate-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.724936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.724879    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-metrics-client-ca\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.724936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.724905    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzf8q\" (UniqueName: \"kubernetes.io/projected/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-kube-api-access-xzf8q\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.724936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.724923    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725164 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.724952    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-serving-certs-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725164 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.724968    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725164 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.725084    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725164 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.725115    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725361 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.725219    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-federate-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.725639 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.725596    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-metrics-client-ca\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.726099 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.726074    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-serving-certs-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.726170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.726111    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.727713 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.727690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-federate-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.727782 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.727736    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.727987 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.727972    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-secret-telemeter-client\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.728104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.728085    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-telemeter-client-tls\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.740003 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.739981    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzf8q\" (UniqueName: \"kubernetes.io/projected/c75ecd60-569f-499c-b1bc-8e3ca2e4632c-kube-api-access-xzf8q\") pod \"telemeter-client-655f6986b-dbx6l\" (UID: \"c75ecd60-569f-499c-b1bc-8e3ca2e4632c\") " pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:00.760184 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:00.760163    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l"
Sep 17 19:08:01.036828 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:01.036776    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-655f6986b-dbx6l"]
Sep 17 19:08:01.038549 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:08:01.038526    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc75ecd60_569f_499c_b1bc_8e3ca2e4632c.slice/crio-eef050805a88b9d55804274cf9041589679fa25951e56f34831473552cbf5f7f WatchSource:0}: Error finding container eef050805a88b9d55804274cf9041589679fa25951e56f34831473552cbf5f7f: Status 404 returned error can't find the container with id eef050805a88b9d55804274cf9041589679fa25951e56f34831473552cbf5f7f
Sep 17 19:08:01.652293 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:01.652255    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l" event={"ID":"c75ecd60-569f-499c-b1bc-8e3ca2e4632c","Type":"ContainerStarted","Data":"eef050805a88b9d55804274cf9041589679fa25951e56f34831473552cbf5f7f"}
Sep 17 19:08:03.662011 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:03.661979    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l" event={"ID":"c75ecd60-569f-499c-b1bc-8e3ca2e4632c","Type":"ContainerStarted","Data":"4a1090b994a91e60de26032655b67483e5bb2e1d318f85b261ebb23f5a8adb78"}
Sep 17 19:08:04.670730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.670687    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l" event={"ID":"c75ecd60-569f-499c-b1bc-8e3ca2e4632c","Type":"ContainerStarted","Data":"014815bf1af00eeb926020e03653b3180a13421cea213d471a5662155d8f1da9"}
Sep 17 19:08:04.671193 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.670768    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l" event={"ID":"c75ecd60-569f-499c-b1bc-8e3ca2e4632c","Type":"ContainerStarted","Data":"1035a16d6dcd98826b7bd8c803c061e90aa0cd816c6b3afd85d11a68fd0ecf22"}
Sep 17 19:08:04.699426 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.699368    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-655f6986b-dbx6l" podStartSLOduration=1.88124976 podStartE2EDuration="4.699344833s" totalImagesPullingTime="2.818095073s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:04.698141242 +0000 UTC m=+129.863882538" watchObservedRunningTime="2026-09-17 19:08:04.699344833 +0000 UTC m=+129.865086110"
Sep 17 19:08:04.807995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.807964    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:04.810339 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.810313    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.824583 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.824558    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:04.875672 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.875553    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:04.933302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.933228    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960467    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960548    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960584    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fmdz\" (UniqueName: \"kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960636    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960696    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960719    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.960887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.960753    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:04.997503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:04.997478    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:05.051768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.051731    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:05.061349 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061322    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061377    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061405    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fmdz\" (UniqueName: \"kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061431    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061689 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061589    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061689 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061659    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.061798 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.061706    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.062171 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.062148    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.062357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.062321    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.062475 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.062420    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.062606 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.062588    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.063939 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.063922    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.064740 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.064722    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.070243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.070224    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fmdz\" (UniqueName: \"kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz\") pod \"console-58cbbb74dd-9rpt8\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") " pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.118585 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.118555    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:05.398963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.398932    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:05.401611 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:08:05.401578    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6872b2b0_7778_4331_ada9_bd5d0036aec4.slice/crio-debd287922eb1727ccd9c590dc0e7056b72febf4fc536d674e3bc2948e37807c WatchSource:0}: Error finding container debd287922eb1727ccd9c590dc0e7056b72febf4fc536d674e3bc2948e37807c: Status 404 returned error can't find the container with id debd287922eb1727ccd9c590dc0e7056b72febf4fc536d674e3bc2948e37807c
Sep 17 19:08:05.603846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.603810    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:05.675785 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.675711    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbbb74dd-9rpt8" event={"ID":"6872b2b0-7778-4331-ada9-bd5d0036aec4","Type":"ContainerStarted","Data":"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"}
Sep 17 19:08:05.676095 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.675792    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbbb74dd-9rpt8" event={"ID":"6872b2b0-7778-4331-ada9-bd5d0036aec4","Type":"ContainerStarted","Data":"debd287922eb1727ccd9c590dc0e7056b72febf4fc536d674e3bc2948e37807c"}
Sep 17 19:08:05.695466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.695415    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58cbbb74dd-9rpt8" podStartSLOduration=1.695400915 podStartE2EDuration="1.695400915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:05.694074286 +0000 UTC m=+130.859815585" watchObservedRunningTime="2026-09-17 19:08:05.695400915 +0000 UTC m=+130.861142189"
Sep 17 19:08:05.699314 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:05.699294    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:10.563960 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:10.563928    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:10.625024 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:10.624997    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:15.119572 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.119528    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:15.120032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.119593    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:15.124293 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.124271    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:15.712823 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.712795    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:15.762400 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.762374    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:08:15.904149 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.904119    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:08:15.906904 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.906888    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.910233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.910207    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:15.921015 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.920993    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:08:15.926984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.926965    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.926994    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.927024    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.927047    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.927080    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.927094    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.927192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.927112    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jxbm\" (UniqueName: \"kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:15.964682 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:15.964610    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:16.027442 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027421    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027516 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027466    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027516 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027485    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027592 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027513    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027592 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027592 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027568    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jxbm\" (UniqueName: \"kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.027763 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.027665    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.028231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.028209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.028329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.028209    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.028329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.028274    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.028329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.028323    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.029988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.029963    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.030164 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.030147    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.037036 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.037013    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jxbm\" (UniqueName: \"kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm\") pod \"console-5d69ffd86f-92sbw\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") " pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.215468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.215404    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:16.425294 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.425265    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:16.488222 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.488200    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:08:16.490434 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:08:16.490403    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda305c7ed_7fbe_4d40_b8b6_17055c72fb57.slice/crio-f699b0b5c3abb1087dacc885c8672d6d15553451a67eb70a16dd66d1d6798ef8 WatchSource:0}: Error finding container f699b0b5c3abb1087dacc885c8672d6d15553451a67eb70a16dd66d1d6798ef8: Status 404 returned error can't find the container with id f699b0b5c3abb1087dacc885c8672d6d15553451a67eb70a16dd66d1d6798ef8
Sep 17 19:08:16.491251 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.491213    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:16.714146 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.714102    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69ffd86f-92sbw" event={"ID":"a305c7ed-7fbe-4d40-b8b6-17055c72fb57","Type":"ContainerStarted","Data":"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"}
Sep 17 19:08:16.714292 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.714156    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69ffd86f-92sbw" event={"ID":"a305c7ed-7fbe-4d40-b8b6-17055c72fb57","Type":"ContainerStarted","Data":"f699b0b5c3abb1087dacc885c8672d6d15553451a67eb70a16dd66d1d6798ef8"}
Sep 17 19:08:16.736680 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:16.736637    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d69ffd86f-92sbw" podStartSLOduration=1.736606202 podStartE2EDuration="1.736606202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:08:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:08:16.734824446 +0000 UTC m=+141.900565721" watchObservedRunningTime="2026-09-17 19:08:16.736606202 +0000 UTC m=+141.902347476"
Sep 17 19:08:22.447188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:22.447140    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:22.504297 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:22.504271    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:26.215978 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.215920    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:26.216342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.216012    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:26.220665 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.220639    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:26.752072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.752040    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:08:26.799428 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.799402    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:26.902933 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.902899    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:26.956320 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:26.956289    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:27.011025 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:27.010948    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:37.060477 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:37.060397    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:37.163321 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:37.163294    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:08:40.781224 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:40.781184    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6cd77f78-2clvc" podUID="fe14cc17-3d41-4f62-be82-27df48afb593" containerName="console" containerID="cri-o://db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525" gracePeriod=15
Sep 17 19:08:41.022940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.022917    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:08:41.087837 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087775    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.087837 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087808    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.087837 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087832    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m22dl\" (UniqueName: \"kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.088072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087875    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.088072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087898    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.088072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087931    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.088072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.087955    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config\") pod \"fe14cc17-3d41-4f62-be82-27df48afb593\" (UID: \"fe14cc17-3d41-4f62-be82-27df48afb593\") "
Sep 17 19:08:41.088479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.088380    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca" (OuterVolumeSpecName: "service-ca") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:41.088479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.088392    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:41.088479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.088417    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:41.088857 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.088517    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config" (OuterVolumeSpecName: "console-config") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:41.089963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.089943    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:41.090057 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.090029    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl" (OuterVolumeSpecName: "kube-api-access-m22dl") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "kube-api-access-m22dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:41.090098 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.090075    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "fe14cc17-3d41-4f62-be82-27df48afb593" (UID: "fe14cc17-3d41-4f62-be82-27df48afb593"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:41.189329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189302    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-oauth-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189325    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-trusted-ca-bundle\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189334    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-service-ca\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189343    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/fe14cc17-3d41-4f62-be82-27df48afb593-console-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189355    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189367    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/fe14cc17-3d41-4f62-be82-27df48afb593-console-oauth-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.189456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.189375    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m22dl\" (UniqueName: \"kubernetes.io/projected/fe14cc17-3d41-4f62-be82-27df48afb593-kube-api-access-m22dl\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:41.805569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.805536    2586 generic.go:356] "Generic (PLEG): container finished" podID="fe14cc17-3d41-4f62-be82-27df48afb593" containerID="db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525" exitCode=0
Sep 17 19:08:41.806005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.805575    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6cd77f78-2clvc"
Sep 17 19:08:41.806005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.805641    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd77f78-2clvc" event={"ID":"fe14cc17-3d41-4f62-be82-27df48afb593","Type":"ContainerDied","Data":"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"}
Sep 17 19:08:41.806005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.805690    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6cd77f78-2clvc" event={"ID":"fe14cc17-3d41-4f62-be82-27df48afb593","Type":"ContainerDied","Data":"2f2d9f4f363c7d74d6dd305e4f7c49a41ba22409faeb57352ec74ff79b4305ac"}
Sep 17 19:08:41.806005 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.805711    2586 scope.go:117] "RemoveContainer" containerID="db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"
Sep 17 19:08:41.813918 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.813902    2586 scope.go:117] "RemoveContainer" containerID="db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"
Sep 17 19:08:41.814197 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:08:41.814177    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525\": container with ID starting with db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525 not found: ID does not exist" containerID="db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"
Sep 17 19:08:41.814292 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.814202    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525"} err="failed to get container status \"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525\": rpc error: code = NotFound desc = could not find container \"db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525\": container with ID starting with db63fe0a17c4c4d94890525c8f016a95cd7f939615bc4e52657209af9bc74525 not found: ID does not exist"
Sep 17 19:08:41.825807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.825785    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:08:41.829763 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:41.829744    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6cd77f78-2clvc"]
Sep 17 19:08:43.014465 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:43.014435    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe14cc17-3d41-4f62-be82-27df48afb593" path="/var/lib/kubelet/pods/fe14cc17-3d41-4f62-be82-27df48afb593/volumes"
Sep 17 19:08:51.823444 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:51.823391    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58cbbb74dd-9rpt8" podUID="6872b2b0-7778-4331-ada9-bd5d0036aec4" containerName="console" containerID="cri-o://f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5" gracePeriod=15
Sep 17 19:08:52.061138 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.061117    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:52.165070 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.164992    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165070 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165026    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6fmdz\" (UniqueName: \"kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165070 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165047    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165070 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165065    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165093    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165122    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165144    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert\") pod \"6872b2b0-7778-4331-ada9-bd5d0036aec4\" (UID: \"6872b2b0-7778-4331-ada9-bd5d0036aec4\") "
Sep 17 19:08:52.165600 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165566    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:52.165600 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config" (OuterVolumeSpecName: "console-config") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:52.165761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165600    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca" (OuterVolumeSpecName: "service-ca") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:52.165761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.165735    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:08:52.167429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.167406    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz" (OuterVolumeSpecName: "kube-api-access-6fmdz") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "kube-api-access-6fmdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:08:52.167429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.167415    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:52.167547 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.167496    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6872b2b0-7778-4331-ada9-bd5d0036aec4" (UID: "6872b2b0-7778-4331-ada9-bd5d0036aec4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:08:52.266367 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266332    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-oauth-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266367 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266360    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266367 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266369    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-oauth-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266367 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266377    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-trusted-ca-bundle\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266385    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fmdz\" (UniqueName: \"kubernetes.io/projected/6872b2b0-7778-4331-ada9-bd5d0036aec4-kube-api-access-6fmdz\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266393    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-service-ca\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.266616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.266401    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6872b2b0-7778-4331-ada9-bd5d0036aec4-console-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:08:52.849849 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.849816    2586 generic.go:356] "Generic (PLEG): container finished" podID="6872b2b0-7778-4331-ada9-bd5d0036aec4" containerID="f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5" exitCode=0
Sep 17 19:08:52.850290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.849864    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbbb74dd-9rpt8"
Sep 17 19:08:52.850290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.849891    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbbb74dd-9rpt8" event={"ID":"6872b2b0-7778-4331-ada9-bd5d0036aec4","Type":"ContainerDied","Data":"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"}
Sep 17 19:08:52.850290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.849930    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbbb74dd-9rpt8" event={"ID":"6872b2b0-7778-4331-ada9-bd5d0036aec4","Type":"ContainerDied","Data":"debd287922eb1727ccd9c590dc0e7056b72febf4fc536d674e3bc2948e37807c"}
Sep 17 19:08:52.850290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.849949    2586 scope.go:117] "RemoveContainer" containerID="f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"
Sep 17 19:08:52.857994 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.857978    2586 scope.go:117] "RemoveContainer" containerID="f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"
Sep 17 19:08:52.858273 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:08:52.858247    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5\": container with ID starting with f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5 not found: ID does not exist" containerID="f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"
Sep 17 19:08:52.858332 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.858285    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5"} err="failed to get container status \"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5\": rpc error: code = NotFound desc = could not find container \"f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5\": container with ID starting with f5146eeae6b4c41eaa283b211d72103fabbc3a816a062c96e8d7039821a80fb5 not found: ID does not exist"
Sep 17 19:08:52.872041 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.872019    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:52.876105 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:52.876087    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58cbbb74dd-9rpt8"]
Sep 17 19:08:53.013584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:08:53.013560    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6872b2b0-7778-4331-ada9-bd5d0036aec4" path="/var/lib/kubelet/pods/6872b2b0-7778-4331-ada9-bd5d0036aec4/volumes"
Sep 17 19:09:22.169111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:22.169070    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:22.268913 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:22.268881    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:22.373823 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:22.373793    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:23.271949 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.271912    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"]
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272197    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe14cc17-3d41-4f62-be82-27df48afb593" containerName="console"
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272215    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe14cc17-3d41-4f62-be82-27df48afb593" containerName="console"
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272232    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6872b2b0-7778-4331-ada9-bd5d0036aec4" containerName="console"
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272241    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6872b2b0-7778-4331-ada9-bd5d0036aec4" containerName="console"
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272300    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6872b2b0-7778-4331-ada9-bd5d0036aec4" containerName="console"
Sep 17 19:09:23.272379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.272310    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe14cc17-3d41-4f62-be82-27df48afb593" containerName="console"
Sep 17 19:09:23.275542 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.275525    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.279043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.279024    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 17 19:09:23.279286 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.279271    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-5f4bp\""
Sep 17 19:09:23.280019 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.280003    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 17 19:09:23.290572 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.290553    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"]
Sep 17 19:09:23.384507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.384484    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.384632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.384513    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.384632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.384572    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcmtt\" (UniqueName: \"kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.485458 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.485432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcmtt\" (UniqueName: \"kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.485568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.485471    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.485568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.485487    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.485814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.485798    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.485853 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.485822    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.494485 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.494463    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcmtt\" (UniqueName: \"kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt\") pod \"1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") " pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.584345 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.584319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:23.857292 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.857218    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"]
Sep 17 19:09:23.860652 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:09:23.860610    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34d17958_60d4_42fe_ba02_cf99d7553239.slice/crio-d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7 WatchSource:0}: Error finding container d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7: Status 404 returned error can't find the container with id d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7
Sep 17 19:09:23.954633 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:23.954584    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerStarted","Data":"d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7"}
Sep 17 19:09:28.327967 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:28.327930    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:28.388210 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:28.388177    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:29.978670 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:29.978641    2586 generic.go:356] "Generic (PLEG): container finished" podID="34d17958-60d4-42fe-ba02-cf99d7553239" containerID="117caac0bfeefde427c9637f5f42d900879f4da33226cf2fc841416960fa34fc" exitCode=0
Sep 17 19:09:29.979033 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:29.978691    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerDied","Data":"117caac0bfeefde427c9637f5f42d900879f4da33226cf2fc841416960fa34fc"}
Sep 17 19:09:32.991277 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:32.991238    2586 generic.go:356] "Generic (PLEG): container finished" podID="34d17958-60d4-42fe-ba02-cf99d7553239" containerID="065e55659178fd303f43bfdf66bba81f8e67fd9119c674f958db11c50344d754" exitCode=0
Sep 17 19:09:32.991759 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:32.991297    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerDied","Data":"065e55659178fd303f43bfdf66bba81f8e67fd9119c674f958db11c50344d754"}
Sep 17 19:09:39.017640 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:39.017597    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerStarted","Data":"f8453abfccc04019316e351e15c298ff15f225a50410302a7caab69c5c8cf45e"}
Sep 17 19:09:39.039551 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:39.039503    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" podStartSLOduration=2.004813093 podStartE2EDuration="16.03948297s" totalImagesPullingTime="14.034669877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:39.038044131 +0000 UTC m=+224.203785431" watchObservedRunningTime="2026-09-17 19:09:39.03948297 +0000 UTC m=+224.205224246"
Sep 17 19:09:40.023883 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:40.023854    2586 generic.go:356] "Generic (PLEG): container finished" podID="34d17958-60d4-42fe-ba02-cf99d7553239" containerID="f8453abfccc04019316e351e15c298ff15f225a50410302a7caab69c5c8cf45e" exitCode=0
Sep 17 19:09:40.024231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:40.023918    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerDied","Data":"f8453abfccc04019316e351e15c298ff15f225a50410302a7caab69c5c8cf45e"}
Sep 17 19:09:41.150306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.150280    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:41.329931 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.329863    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util\") pod \"34d17958-60d4-42fe-ba02-cf99d7553239\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") "
Sep 17 19:09:41.329931 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.329917    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle\") pod \"34d17958-60d4-42fe-ba02-cf99d7553239\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") "
Sep 17 19:09:41.330086 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.329940    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcmtt\" (UniqueName: \"kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt\") pod \"34d17958-60d4-42fe-ba02-cf99d7553239\" (UID: \"34d17958-60d4-42fe-ba02-cf99d7553239\") "
Sep 17 19:09:41.330568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.330504    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle" (OuterVolumeSpecName: "bundle") pod "34d17958-60d4-42fe-ba02-cf99d7553239" (UID: "34d17958-60d4-42fe-ba02-cf99d7553239"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:41.332228 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.332199    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt" (OuterVolumeSpecName: "kube-api-access-mcmtt") pod "34d17958-60d4-42fe-ba02-cf99d7553239" (UID: "34d17958-60d4-42fe-ba02-cf99d7553239"). InnerVolumeSpecName "kube-api-access-mcmtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:09:41.334432 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.334410    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util" (OuterVolumeSpecName: "util") pod "34d17958-60d4-42fe-ba02-cf99d7553239" (UID: "34d17958-60d4-42fe-ba02-cf99d7553239"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:09:41.430804 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.430776    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcmtt\" (UniqueName: \"kubernetes.io/projected/34d17958-60d4-42fe-ba02-cf99d7553239-kube-api-access-mcmtt\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:09:41.430804 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.430798    2586 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-util\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:09:41.430804 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:41.430808    2586 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/34d17958-60d4-42fe-ba02-cf99d7553239-bundle\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:09:42.032589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:42.032554    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh" event={"ID":"34d17958-60d4-42fe-ba02-cf99d7553239","Type":"ContainerDied","Data":"d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7"}
Sep 17 19:09:42.032589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:42.032576    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/1acd376c2f780381bc434f5f1985f786a96c8309f7a005b52ed03f0c1a2mgfh"
Sep 17 19:09:42.032851 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:42.032597    2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0a7d6df30938636cb7f021e0f3ba86d6cda51e4f4d287ad1e71c6ea21782ec7"
Sep 17 19:09:44.472906 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:44.472873    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:44.589853 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:44.589789    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:46.095786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.095747    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"]
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096084    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="util"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096100    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="util"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096119    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="pull"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096128    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="pull"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096138    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="extract"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096146    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="extract"
Sep 17 19:09:46.096241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.096209    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="34d17958-60d4-42fe-ba02-cf99d7553239" containerName="extract"
Sep 17 19:09:46.130239 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.130216    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"]
Sep 17 19:09:46.130398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.130319    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.132868 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.132846    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-mbwj6\""
Sep 17 19:09:46.132996 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.132891    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Sep 17 19:09:46.132996 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.132851    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Sep 17 19:09:46.132996 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.132938    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Sep 17 19:09:46.158482 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.158453    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qtpw\" (UniqueName: \"kubernetes.io/projected/48034166-01be-4ef1-999f-d31d0ad121d5-kube-api-access-7qtpw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.158591 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.158489    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/48034166-01be-4ef1-999f-d31d0ad121d5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.259265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.259236    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qtpw\" (UniqueName: \"kubernetes.io/projected/48034166-01be-4ef1-999f-d31d0ad121d5-kube-api-access-7qtpw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.259375 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.259276    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/48034166-01be-4ef1-999f-d31d0ad121d5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.261719 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.261697    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/48034166-01be-4ef1-999f-d31d0ad121d5-certificates\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.273211 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.273189    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qtpw\" (UniqueName: \"kubernetes.io/projected/48034166-01be-4ef1-999f-d31d0ad121d5-kube-api-access-7qtpw\") pod \"custom-metrics-autoscaler-operator-566c9cd69c-lnfw7\" (UID: \"48034166-01be-4ef1-999f-d31d0ad121d5\") " pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.439993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.439938    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:46.736033 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:46.736008    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"]
Sep 17 19:09:46.737959 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:09:46.737927    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48034166_01be_4ef1_999f_d31d0ad121d5.slice/crio-4323429c5cff3e70bbfbe8e05b8df8c72d80b9655ee8ae7cb3f593f690acae75 WatchSource:0}: Error finding container 4323429c5cff3e70bbfbe8e05b8df8c72d80b9655ee8ae7cb3f593f690acae75: Status 404 returned error can't find the container with id 4323429c5cff3e70bbfbe8e05b8df8c72d80b9655ee8ae7cb3f593f690acae75
Sep 17 19:09:47.049559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:47.049525    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7" event={"ID":"48034166-01be-4ef1-999f-d31d0ad121d5","Type":"ContainerStarted","Data":"4323429c5cff3e70bbfbe8e05b8df8c72d80b9655ee8ae7cb3f593f690acae75"}
Sep 17 19:09:50.801523 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:50.801491    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:51.066956 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.066877    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7" event={"ID":"48034166-01be-4ef1-999f-d31d0ad121d5","Type":"ContainerStarted","Data":"6e56c9343d1610d2baa79c349548ee0ecf3f453683de6545352e4f1f1fc1e142"}
Sep 17 19:09:51.067090 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.067014    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:09:51.071792 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.071767    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:51.085840 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.085821    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-579c8958f7-spkbx"]
Sep 17 19:09:51.088818 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.088776    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7" podStartSLOduration=1.565738249 podStartE2EDuration="5.088763157s" totalImagesPullingTime="3.523024908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:51.087761472 +0000 UTC m=+236.253502769" watchObservedRunningTime="2026-09-17 19:09:51.088763157 +0000 UTC m=+236.254504432"
Sep 17 19:09:51.108654 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.108607    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-spkbx"]
Sep 17 19:09:51.108772 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.108726    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.111223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.111196    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Sep 17 19:09:51.111223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.111218    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Sep 17 19:09:51.111392 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.111197    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-2x6rm\""
Sep 17 19:09:51.188074 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.188044    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:51.188194 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.188120    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shgcg\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-kube-api-access-shgcg\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.188251 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.188202    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.188308 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.188250    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-cabundle0\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.288840 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.288815    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shgcg\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-kube-api-access-shgcg\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.288995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.288863    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.288995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.288915    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-cabundle0\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.289283 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.289265    2586 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:09:51.289283 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.289285    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:09:51.289430 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.289293    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-spkbx: references non-existent secret key: ca.crt
Sep 17 19:09:51.289430 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.289357    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates podName:4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723 nodeName:}" failed. No retries permitted until 2026-09-17 19:09:51.789340105 +0000 UTC m=+236.955081364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates") pod "keda-operator-579c8958f7-spkbx" (UID: "4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723") : references non-existent secret key: ca.crt
Sep 17 19:09:51.289753 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.289734    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-cabundle0\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.289825 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.289793    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:51.297431 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.297408    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shgcg\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-kube-api-access-shgcg\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.356706 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.356655    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"]
Sep 17 19:09:51.377852 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.377834    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"]
Sep 17 19:09:51.377922 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.377912    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.380206 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.380189    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Sep 17 19:09:51.389139 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.389118    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgt6\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-kube-api-access-bfgt6\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.389236 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.389150    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5d540fbe-67d3-4310-90bb-676866f3fa99-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.389236 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.389196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.470665 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.470631    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:51.490226 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.490205    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.490318 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.490242    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgt6\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-kube-api-access-bfgt6\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.490318 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.490290    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5d540fbe-67d3-4310-90bb-676866f3fa99-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.490421 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.490349    2586 secret.go:281] references non-existent secret key: tls.crt
Sep 17 19:09:51.490421 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.490361    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 19:09:51.490421 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.490376    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r: references non-existent secret key: tls.crt
Sep 17 19:09:51.490563 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.490433    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates podName:5d540fbe-67d3-4310-90bb-676866f3fa99 nodeName:}" failed. No retries permitted until 2026-09-17 19:09:51.990416793 +0000 UTC m=+237.156158046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates") pod "keda-metrics-apiserver-77857b7b98-lsm8r" (UID: "5d540fbe-67d3-4310-90bb-676866f3fa99") : references non-existent secret key: tls.crt
Sep 17 19:09:51.490634 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.490576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/5d540fbe-67d3-4310-90bb-676866f3fa99-temp-vol\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.503986 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.503960    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgt6\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-kube-api-access-bfgt6\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.791582 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.791556    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:51.791728 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.791714    2586 secret.go:281] references non-existent secret key: ca.crt
Sep 17 19:09:51.791775 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.791733    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Sep 17 19:09:51.791775 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.791741    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-579c8958f7-spkbx: references non-existent secret key: ca.crt
Sep 17 19:09:51.791837 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.791788    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates podName:4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723 nodeName:}" failed. No retries permitted until 2026-09-17 19:09:52.791771812 +0000 UTC m=+237.957513066 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates") pod "keda-operator-579c8958f7-spkbx" (UID: "4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723") : references non-existent secret key: ca.crt
Sep 17 19:09:51.992841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:51.992813    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:51.993193 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.992928    2586 secret.go:281] references non-existent secret key: tls.crt
Sep 17 19:09:51.993193 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.992942    2586 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Sep 17 19:09:51.993193 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.992959    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r: references non-existent secret key: tls.crt
Sep 17 19:09:51.993193 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:51.993005    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates podName:5d540fbe-67d3-4310-90bb-676866f3fa99 nodeName:}" failed. No retries permitted until 2026-09-17 19:09:52.992993284 +0000 UTC m=+238.158734537 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates") pod "keda-metrics-apiserver-77857b7b98-lsm8r" (UID: "5d540fbe-67d3-4310-90bb-676866f3fa99") : references non-existent secret key: tls.crt
Sep 17 19:09:52.051040 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.050964    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.060668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.060645    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f8fmb"]
Sep 17 19:09:52.081136 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.081115    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f8fmb"]
Sep 17 19:09:52.081244 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.081207    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.083677 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.083650    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Sep 17 19:09:52.093112 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.093090    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.093210 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.093162    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l6rg\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-kube-api-access-5l6rg\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.110058 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.110035    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.193917 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.193872    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.194058 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.193965    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l6rg\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-kube-api-access-5l6rg\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.194058 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:52.194008    2586 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Sep 17 19:09:52.194058 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:52.194032    2586 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-6c4d864b86-f8fmb: secret "keda-admission-webhooks-certs" not found
Sep 17 19:09:52.194202 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:09:52.194077    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates podName:409d0b1c-f3b2-452e-8780-95049c87b9e9 nodeName:}" failed. No retries permitted until 2026-09-17 19:09:52.694062394 +0000 UTC m=+237.859803647 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates") pod "keda-admission-6c4d864b86-f8fmb" (UID: "409d0b1c-f3b2-452e-8780-95049c87b9e9") : secret "keda-admission-webhooks-certs" not found
Sep 17 19:09:52.202830 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.202804    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l6rg\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-kube-api-access-5l6rg\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.573672 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.573645    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.671985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.671957    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.697176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.697148    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.699667 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.699651    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/409d0b1c-f3b2-452e-8780-95049c87b9e9-certificates\") pod \"keda-admission-6c4d864b86-f8fmb\" (UID: \"409d0b1c-f3b2-452e-8780-95049c87b9e9\") " pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.797921 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.797898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:52.800272 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.800255    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723-certificates\") pod \"keda-operator-579c8958f7-spkbx\" (UID: \"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723\") " pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:52.833609 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.833540    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.918032 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.918005    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:52.991994 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.991964    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:52.998029 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.997994    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:52.999237 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:52.999213    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:53.002173 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.002155    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/5d540fbe-67d3-4310-90bb-676866f3fa99-certificates\") pod \"keda-metrics-apiserver-77857b7b98-lsm8r\" (UID: \"5d540fbe-67d3-4310-90bb-676866f3fa99\") " pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:53.105132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.105071    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:53.187846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.187819    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:53.191895 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.191877    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-579c8958f7-spkbx"]
Sep 17 19:09:53.195417 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:09:53.195391    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd9c6dd_ecb1_49b0_b2ef_fec06fccc723.slice/crio-732d5c060d7ec82573c569203ebd8b0514aef124535990d7056a62e55c355cd4 WatchSource:0}: Error finding container 732d5c060d7ec82573c569203ebd8b0514aef124535990d7056a62e55c355cd4: Status 404 returned error can't find the container with id 732d5c060d7ec82573c569203ebd8b0514aef124535990d7056a62e55c355cd4
Sep 17 19:09:53.230610 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.230582    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:09:53.262768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.262692    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-6c4d864b86-f8fmb"]
Sep 17 19:09:53.265090 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:09:53.265063    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod409d0b1c_f3b2_452e_8780_95049c87b9e9.slice/crio-4ae078e22497055caf5829af7e35174ec1e600aa481441d271bd17aa5f65ef0b WatchSource:0}: Error finding container 4ae078e22497055caf5829af7e35174ec1e600aa481441d271bd17aa5f65ef0b: Status 404 returned error can't find the container with id 4ae078e22497055caf5829af7e35174ec1e600aa481441d271bd17aa5f65ef0b
Sep 17 19:09:53.457870 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:53.457814    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"]
Sep 17 19:09:53.459828 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:09:53.459798    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d540fbe_67d3_4310_90bb_676866f3fa99.slice/crio-d7d4df9a4a5ce44514140f21b959c18c573912103116bf6927d22a605f5a85e3 WatchSource:0}: Error finding container d7d4df9a4a5ce44514140f21b959c18c573912103116bf6927d22a605f5a85e3: Status 404 returned error can't find the container with id d7d4df9a4a5ce44514140f21b959c18c573912103116bf6927d22a605f5a85e3
Sep 17 19:09:54.079428 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:54.079386    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb" event={"ID":"409d0b1c-f3b2-452e-8780-95049c87b9e9","Type":"ContainerStarted","Data":"4ae078e22497055caf5829af7e35174ec1e600aa481441d271bd17aa5f65ef0b"}
Sep 17 19:09:54.080795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:54.080760    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r" event={"ID":"5d540fbe-67d3-4310-90bb-676866f3fa99","Type":"ContainerStarted","Data":"d7d4df9a4a5ce44514140f21b959c18c573912103116bf6927d22a605f5a85e3"}
Sep 17 19:09:54.082086 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:54.082064    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-spkbx" event={"ID":"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723","Type":"ContainerStarted","Data":"732d5c060d7ec82573c569203ebd8b0514aef124535990d7056a62e55c355cd4"}
Sep 17 19:09:56.094489 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:56.094442    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb" event={"ID":"409d0b1c-f3b2-452e-8780-95049c87b9e9","Type":"ContainerStarted","Data":"4b86eeb8d59cbe09f23e39ca7391b6ceb6e6577555aaf5535fc341b7da6843a6"}
Sep 17 19:09:56.094988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:56.094785    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:09:58.105046 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.105002    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r" event={"ID":"5d540fbe-67d3-4310-90bb-676866f3fa99","Type":"ContainerStarted","Data":"f5eba515916c2ff6cf3d7f2f48c83ae1962228036a35277e961abfeab5949ca2"}
Sep 17 19:09:58.105503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.105123    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:09:58.106841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.106818    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-579c8958f7-spkbx" event={"ID":"4bd9c6dd-ecb1-49b0-b2ef-fec06fccc723","Type":"ContainerStarted","Data":"e4a9aa4a5a39f72fb5b7c0a6324dc734b7c675b99dc7b30d0d832938b809e607"}
Sep 17 19:09:58.106964 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.106874    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:09:58.122709 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.122675    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb" podStartSLOduration=4.151427053 podStartE2EDuration="6.122661183s" totalImagesPullingTime="1.97123413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:56.110761909 +0000 UTC m=+241.276503205" watchObservedRunningTime="2026-09-17 19:09:58.122661183 +0000 UTC m=+243.288402615"
Sep 17 19:09:58.122999 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.122970    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r" podStartSLOduration=3.416068038 podStartE2EDuration="7.122965023s" totalImagesPullingTime="3.706896985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:58.121649697 +0000 UTC m=+243.287390996" watchObservedRunningTime="2026-09-17 19:09:58.122965023 +0000 UTC m=+243.288706295"
Sep 17 19:09:58.137138 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:09:58.137096    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-579c8958f7-spkbx" podStartSLOduration=3.165404091 podStartE2EDuration="7.137081857s" totalImagesPullingTime="3.971677766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:09:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:09:58.135782574 +0000 UTC m=+243.301523870" watchObservedRunningTime="2026-09-17 19:09:58.137081857 +0000 UTC m=+243.302823131"
Sep 17 19:10:09.115280 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:09.115246    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-77857b7b98-lsm8r"
Sep 17 19:10:12.072167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:12.072136    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-566c9cd69c-lnfw7"
Sep 17 19:10:17.101528 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:17.101498    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-6c4d864b86-f8fmb"
Sep 17 19:10:19.113899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:19.113871    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-579c8958f7-spkbx"
Sep 17 19:10:43.204277 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:43.204248    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:44.409640 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:44.409594    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:10:56.925224 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:56.925194    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:10:56.927303 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:56.927278    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:10:56.928436 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:10:56.928412    2586 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 19:11:08.727806 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:08.727770    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:11.793085 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.793052    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:11.856903 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.856878    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:11.858857 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.858842    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:11.861278 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.861254    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Sep 17 19:11:11.861435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.861292    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Sep 17 19:11:11.861435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.861380    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-kdwvb\""
Sep 17 19:11:11.861553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.861416    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:11.862305 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.862290    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Sep 17 19:11:11.864995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.864975    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"]
Sep 17 19:11:11.866864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.866846    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:11.869009 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.868991    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-webhook-server-cert\""
Sep 17 19:11:11.869107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.869009    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"llmisvc-controller-manager-dockercfg-78dgr\""
Sep 17 19:11:11.871930 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.871910    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:11.883521 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.883494    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"]
Sep 17 19:11:11.936006 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.935965    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:11.967343 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.967315    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5klt2\" (UniqueName: \"kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:11.967468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.967349    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdrm5\" (UniqueName: \"kubernetes.io/projected/8183ffec-cca1-4146-bb29-2128a07e952f-kube-api-access-gdrm5\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:11.967468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.967413    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8183ffec-cca1-4146-bb29-2128a07e952f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:11.967468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.967447    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:11.999207 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:11.999163    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:12.067880 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.067819    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5klt2\" (UniqueName: \"kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.067880 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.067851    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdrm5\" (UniqueName: \"kubernetes.io/projected/8183ffec-cca1-4146-bb29-2128a07e952f-kube-api-access-gdrm5\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:12.067880 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.067876    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8183ffec-cca1-4146-bb29-2128a07e952f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:12.068054 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.067898    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.068054 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:11:12.067992    2586 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 17 19:11:12.068054 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:11:12.068044    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert podName:aac6d5e8-d927-468b-a941-853321ed9316 nodeName:}" failed. No retries permitted until 2026-09-17 19:11:12.568027476 +0000 UTC m=+317.733768745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert") pod "kserve-controller-manager-66576f5db6-ms8tt" (UID: "aac6d5e8-d927-468b-a941-853321ed9316") : secret "kserve-webhook-server-cert" not found
Sep 17 19:11:12.070383 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.070366    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8183ffec-cca1-4146-bb29-2128a07e952f-cert\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:12.096111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.096081    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5klt2\" (UniqueName: \"kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.097276 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.097254    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdrm5\" (UniqueName: \"kubernetes.io/projected/8183ffec-cca1-4146-bb29-2128a07e952f-kube-api-access-gdrm5\") pod \"llmisvc-controller-manager-645f9c8fd6-46rp7\" (UID: \"8183ffec-cca1-4146-bb29-2128a07e952f\") " pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:12.147450 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.147415    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:12.177761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.177737    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:12.447955 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.447933    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"]
Sep 17 19:11:12.450129 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:11:12.450106    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8183ffec_cca1_4146_bb29_2128a07e952f.slice/crio-423de281059a70d1d6db2e17da76e43269a70514e22f2f91bffe3f3d1dbe76d6 WatchSource:0}: Error finding container 423de281059a70d1d6db2e17da76e43269a70514e22f2f91bffe3f3d1dbe76d6: Status 404 returned error can't find the container with id 423de281059a70d1d6db2e17da76e43269a70514e22f2f91bffe3f3d1dbe76d6
Sep 17 19:11:12.451283 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.451264    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:11:12.570811 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.570787    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.573108 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.573091    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") pod \"kserve-controller-manager-66576f5db6-ms8tt\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") " pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.714983 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.714925    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:12.769384 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.769361    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:12.819459 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.819406    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:12.916985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:12.916956    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:11:13.039282 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:13.039214    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:13.042445 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:11:13.042416    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac6d5e8_d927_468b_a941_853321ed9316.slice/crio-536662b314c4ca0543783c0300aa8476b6f8bfff5080390980955415f5580b0a WatchSource:0}: Error finding container 536662b314c4ca0543783c0300aa8476b6f8bfff5080390980955415f5580b0a: Status 404 returned error can't find the container with id 536662b314c4ca0543783c0300aa8476b6f8bfff5080390980955415f5580b0a
Sep 17 19:11:13.425081 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:13.424995    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7" event={"ID":"8183ffec-cca1-4146-bb29-2128a07e952f","Type":"ContainerStarted","Data":"423de281059a70d1d6db2e17da76e43269a70514e22f2f91bffe3f3d1dbe76d6"}
Sep 17 19:11:13.426681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:13.426651    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" event={"ID":"aac6d5e8-d927-468b-a941-853321ed9316","Type":"ContainerStarted","Data":"536662b314c4ca0543783c0300aa8476b6f8bfff5080390980955415f5580b0a"}
Sep 17 19:11:16.442500 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.442461    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7" event={"ID":"8183ffec-cca1-4146-bb29-2128a07e952f","Type":"ContainerStarted","Data":"7acd80fdceb13ab4a858e15ba812d821353ca483750023cc963c391513e888ed"}
Sep 17 19:11:16.442960 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.442530    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:16.443976 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.443954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" event={"ID":"aac6d5e8-d927-468b-a941-853321ed9316","Type":"ContainerStarted","Data":"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"}
Sep 17 19:11:16.444051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.444010    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:16.459532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.459488    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7" podStartSLOduration=2.234922064 podStartE2EDuration="5.459472025s" totalImagesPullingTime="3.224549961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:16.458751103 +0000 UTC m=+321.624492418" watchObservedRunningTime="2026-09-17 19:11:16.459472025 +0000 UTC m=+321.625213302"
Sep 17 19:11:16.474149 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:16.474107    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" podStartSLOduration=2.8402814960000002 podStartE2EDuration="5.474089037s" totalImagesPullingTime="2.633807541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:16.473595858 +0000 UTC m=+321.639337210" watchObservedRunningTime="2026-09-17 19:11:16.474089037 +0000 UTC m=+321.639830375"
Sep 17 19:11:47.451138 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:47.451060    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/llmisvc-controller-manager-645f9c8fd6-46rp7"
Sep 17 19:11:47.453837 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:47.453820    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:56.951369 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:56.951336    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:56.951790 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:56.951525    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" podUID="aac6d5e8-d927-468b-a941-853321ed9316" containerName="manager" containerID="cri-o://57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0" gracePeriod=10
Sep 17 19:11:56.976247 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:56.976225    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-sgsxc"]
Sep 17 19:11:56.978100 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:56.978085    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:56.991493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:56.991468    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-sgsxc"]
Sep 17 19:11:57.086021 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.085984    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/388856fa-d8f4-455b-8035-8e8a30d464d1-cert\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.086174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.086065    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnzqx\" (UniqueName: \"kubernetes.io/projected/388856fa-d8f4-455b-8035-8e8a30d464d1-kube-api-access-mnzqx\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.179268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.179244    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:57.187259 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.187237    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnzqx\" (UniqueName: \"kubernetes.io/projected/388856fa-d8f4-455b-8035-8e8a30d464d1-kube-api-access-mnzqx\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.187335 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.187323    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/388856fa-d8f4-455b-8035-8e8a30d464d1-cert\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.189638 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.189602    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/388856fa-d8f4-455b-8035-8e8a30d464d1-cert\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.196923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.196897    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnzqx\" (UniqueName: \"kubernetes.io/projected/388856fa-d8f4-455b-8035-8e8a30d464d1-kube-api-access-mnzqx\") pod \"kserve-controller-manager-66576f5db6-sgsxc\" (UID: \"388856fa-d8f4-455b-8035-8e8a30d464d1\") " pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.288574 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.288547    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") pod \"aac6d5e8-d927-468b-a941-853321ed9316\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") "
Sep 17 19:11:57.288743 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.288614    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5klt2\" (UniqueName: \"kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2\") pod \"aac6d5e8-d927-468b-a941-853321ed9316\" (UID: \"aac6d5e8-d927-468b-a941-853321ed9316\") "
Sep 17 19:11:57.290874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.290843    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert" (OuterVolumeSpecName: "cert") pod "aac6d5e8-d927-468b-a941-853321ed9316" (UID: "aac6d5e8-d927-468b-a941-853321ed9316"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:11:57.290983 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.290907    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2" (OuterVolumeSpecName: "kube-api-access-5klt2") pod "aac6d5e8-d927-468b-a941-853321ed9316" (UID: "aac6d5e8-d927-468b-a941-853321ed9316"). InnerVolumeSpecName "kube-api-access-5klt2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:11:57.355876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.355849    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:57.390021 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.389992    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5klt2\" (UniqueName: \"kubernetes.io/projected/aac6d5e8-d927-468b-a941-853321ed9316-kube-api-access-5klt2\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:11:57.390021 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.390015    2586 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/aac6d5e8-d927-468b-a941-853321ed9316-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:11:57.609204 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.609122    2586 generic.go:356] "Generic (PLEG): container finished" podID="aac6d5e8-d927-468b-a941-853321ed9316" containerID="57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0" exitCode=0
Sep 17 19:11:57.609204 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.609170    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt"
Sep 17 19:11:57.609483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.609171    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" event={"ID":"aac6d5e8-d927-468b-a941-853321ed9316","Type":"ContainerDied","Data":"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"}
Sep 17 19:11:57.609483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.609293    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-ms8tt" event={"ID":"aac6d5e8-d927-468b-a941-853321ed9316","Type":"ContainerDied","Data":"536662b314c4ca0543783c0300aa8476b6f8bfff5080390980955415f5580b0a"}
Sep 17 19:11:57.609483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.609324    2586 scope.go:117] "RemoveContainer" containerID="57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"
Sep 17 19:11:57.619737 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.619717    2586 scope.go:117] "RemoveContainer" containerID="57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"
Sep 17 19:11:57.620010 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:11:57.619993    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0\": container with ID starting with 57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0 not found: ID does not exist" containerID="57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"
Sep 17 19:11:57.620073 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.620018    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0"} err="failed to get container status \"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0\": rpc error: code = NotFound desc = could not find container \"57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0\": container with ID starting with 57542a9fdcec9a0ae5009de3b9672c09500a50b12e01e148735a844b4b5325f0 not found: ID does not exist"
Sep 17 19:11:57.624150 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.624131    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-sgsxc"]
Sep 17 19:11:57.626052 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:11:57.626033    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388856fa_d8f4_455b_8035_8e8a30d464d1.slice/crio-0f03ff31bd14725bd4ae23d621411ac7dbdb98bfc65bec1fc4d6f5c27b8f39e0 WatchSource:0}: Error finding container 0f03ff31bd14725bd4ae23d621411ac7dbdb98bfc65bec1fc4d6f5c27b8f39e0: Status 404 returned error can't find the container with id 0f03ff31bd14725bd4ae23d621411ac7dbdb98bfc65bec1fc4d6f5c27b8f39e0
Sep 17 19:11:57.631088 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.631070    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:57.636681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:57.636662    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-66576f5db6-ms8tt"]
Sep 17 19:11:58.615576 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:58.615526    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc" event={"ID":"388856fa-d8f4-455b-8035-8e8a30d464d1","Type":"ContainerStarted","Data":"2ece556f80ff8c57f7111490a1c817864462199cbd2c8fb11fa1995c47635665"}
Sep 17 19:11:58.616038 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:58.615648    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc" event={"ID":"388856fa-d8f4-455b-8035-8e8a30d464d1","Type":"ContainerStarted","Data":"0f03ff31bd14725bd4ae23d621411ac7dbdb98bfc65bec1fc4d6f5c27b8f39e0"}
Sep 17 19:11:58.616038 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:58.615676    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:11:58.632728 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:58.632680    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc" podStartSLOduration=2.22163329 podStartE2EDuration="2.632661457s" totalImagesPullingTime="411.028167ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:11:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:11:58.631303059 +0000 UTC m=+363.797044348" watchObservedRunningTime="2026-09-17 19:11:58.632661457 +0000 UTC m=+363.798402733"
Sep 17 19:11:59.014605 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:11:59.014531    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac6d5e8-d927-468b-a941-853321ed9316" path="/var/lib/kubelet/pods/aac6d5e8-d927-468b-a941-853321ed9316/volumes"
Sep 17 19:12:00.833698 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.833661    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6445cdb596-2k996"]
Sep 17 19:12:00.834158 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.834064    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac6d5e8-d927-468b-a941-853321ed9316" containerName="manager"
Sep 17 19:12:00.834158 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.834081    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac6d5e8-d927-468b-a941-853321ed9316" containerName="manager"
Sep 17 19:12:00.834284 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.834160    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac6d5e8-d927-468b-a941-853321ed9316" containerName="manager"
Sep 17 19:12:00.835929 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.835909    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.848948 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.848927    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6445cdb596-2k996"]
Sep 17 19:12:00.889552 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.889529    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:00.914652 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914612    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-oauth-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914717 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914656    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-service-ca\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914717 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914681    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-trusted-ca-bundle\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914792 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914717    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914792 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914755    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-oauth-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914792 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914784    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffkt\" (UniqueName: \"kubernetes.io/projected/6468cb42-b2d1-496e-8c72-4f968a63b948-kube-api-access-nffkt\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.914882 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.914812    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-console-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:00.949116 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:00.949088    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:01.015157 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015134    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-oauth-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015166    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-service-ca\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015181    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-trusted-ca-bundle\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015208    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015251    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-oauth-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015462 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffkt\" (UniqueName: \"kubernetes.io/projected/6468cb42-b2d1-496e-8c72-4f968a63b948-kube-api-access-nffkt\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015462 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015327    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-console-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.015988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.015959    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-service-ca\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.016174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.016099    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-console-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.016174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.016128    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-oauth-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.016174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.016159    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6468cb42-b2d1-496e-8c72-4f968a63b948-trusted-ca-bundle\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.017818 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.017799    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-serving-cert\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.017910 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.017821    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6468cb42-b2d1-496e-8c72-4f968a63b948-console-oauth-config\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.025072 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.025047    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffkt\" (UniqueName: \"kubernetes.io/projected/6468cb42-b2d1-496e-8c72-4f968a63b948-kube-api-access-nffkt\") pod \"console-6445cdb596-2k996\" (UID: \"6468cb42-b2d1-496e-8c72-4f968a63b948\") " pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.144306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.144252    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:01.417667 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.417643    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6445cdb596-2k996"]
Sep 17 19:12:01.419521 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:12:01.419502    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6468cb42_b2d1_496e_8c72_4f968a63b948.slice/crio-ca0eac1526085a0ab3425559c9a38cba15dcfb9ff2dede21d4e386e03d680654 WatchSource:0}: Error finding container ca0eac1526085a0ab3425559c9a38cba15dcfb9ff2dede21d4e386e03d680654: Status 404 returned error can't find the container with id ca0eac1526085a0ab3425559c9a38cba15dcfb9ff2dede21d4e386e03d680654
Sep 17 19:12:01.628393 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.628364    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6445cdb596-2k996" event={"ID":"6468cb42-b2d1-496e-8c72-4f968a63b948","Type":"ContainerStarted","Data":"24c6bdb951541a3c0d2b4b61d716e4d0b686a2ec34bee21e4fe12abaa8a15752"}
Sep 17 19:12:01.628558 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.628408    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6445cdb596-2k996" event={"ID":"6468cb42-b2d1-496e-8c72-4f968a63b948","Type":"ContainerStarted","Data":"ca0eac1526085a0ab3425559c9a38cba15dcfb9ff2dede21d4e386e03d680654"}
Sep 17 19:12:01.647598 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:01.647555    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6445cdb596-2k996" podStartSLOduration=1.647541188 podStartE2EDuration="1.647541188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:01.646087262 +0000 UTC m=+366.811828555" watchObservedRunningTime="2026-09-17 19:12:01.647541188 +0000 UTC m=+366.813282463"
Sep 17 19:12:11.145027 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.144985    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:11.145027 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.145040    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:11.149653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.149633    2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:11.673055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.673022    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6445cdb596-2k996"
Sep 17 19:12:11.725017 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.724980    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:12:11.824204 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.824175    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:11.879055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.879028    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:11.939395 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:11.939322    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:29.624548 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:29.624519    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-66576f5db6-sgsxc"
Sep 17 19:12:30.951457 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:30.951427    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:30.993082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:30.993055    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-nhfvm"]
Sep 17 19:12:30.996414 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:30.996394    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:30.999205 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:30.999182    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Sep 17 19:12:30.999205 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:30.999194    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-4pz64\""
Sep 17 19:12:31.009871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.009848    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-nhfvm"]
Sep 17 19:12:31.016410 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.016389    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-7b966d7984-cgwwt"]
Sep 17 19:12:31.019662 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.019644    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.022373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.022353    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-metrics-tls\""
Sep 17 19:12:31.022373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.022357    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Sep 17 19:12:31.022670 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.022391    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-jmp7t\""
Sep 17 19:12:31.024893 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.024874    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cgwwt"]
Sep 17 19:12:31.027129 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.027096    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:31.116487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.116458    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:31.137150 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.137127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032610a4-c869-474a-beff-9748c519be5f-tls-certs\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.137249 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.137169    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc8xq\" (UniqueName: \"kubernetes.io/projected/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-kube-api-access-dc8xq\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.137249 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.137196    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-cert\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.137249 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.137240    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nc6x\" (UniqueName: \"kubernetes.io/projected/032610a4-c869-474a-beff-9748c519be5f-kube-api-access-4nc6x\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.137364 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.137286    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-metrics-certs\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.218168 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.218104    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:31.237810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.237786    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc8xq\" (UniqueName: \"kubernetes.io/projected/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-kube-api-access-dc8xq\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.237913 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.237837    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-cert\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.237913 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.237868    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nc6x\" (UniqueName: \"kubernetes.io/projected/032610a4-c869-474a-beff-9748c519be5f-kube-api-access-4nc6x\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.238023 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.237919    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-metrics-certs\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.238023 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.237946    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032610a4-c869-474a-beff-9748c519be5f-tls-certs\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.240596 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.240576    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/032610a4-c869-474a-beff-9748c519be5f-tls-certs\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.240766 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.240747    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-cert\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.240811 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.240747    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-metrics-certs\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.245719 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.245693    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nc6x\" (UniqueName: \"kubernetes.io/projected/032610a4-c869-474a-beff-9748c519be5f-kube-api-access-4nc6x\") pod \"model-serving-api-7b954ffc95-nhfvm\" (UID: \"032610a4-c869-474a-beff-9748c519be5f\") " pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.245841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.245825    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc8xq\" (UniqueName: \"kubernetes.io/projected/05e5e562-7740-49e7-acdf-6e4bbfd6a9fd-kube-api-access-dc8xq\") pod \"odh-model-controller-7b966d7984-cgwwt\" (UID: \"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd\") " pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.306814 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.306785    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:31.316905 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.316861    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:31.329761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.329737    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:31.581472 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.581450    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-nhfvm"]
Sep 17 19:12:31.583504 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:12:31.583477    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032610a4_c869_474a_beff_9748c519be5f.slice/crio-7e65264049c91061f5b8825b7342468d846305625cc48f7253a0d79194ba3ab3 WatchSource:0}: Error finding container 7e65264049c91061f5b8825b7342468d846305625cc48f7253a0d79194ba3ab3: Status 404 returned error can't find the container with id 7e65264049c91061f5b8825b7342468d846305625cc48f7253a0d79194ba3ab3
Sep 17 19:12:31.607697 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.607596    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-7b966d7984-cgwwt"]
Sep 17 19:12:31.609402 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:12:31.609384    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e5e562_7740_49e7_acdf_6e4bbfd6a9fd.slice/crio-44ccb2848d4eebc1ef5d222c6a4d79c0faec80f46bfb5f827bb8ed9c384f973b WatchSource:0}: Error finding container 44ccb2848d4eebc1ef5d222c6a4d79c0faec80f46bfb5f827bb8ed9c384f973b: Status 404 returned error can't find the container with id 44ccb2848d4eebc1ef5d222c6a4d79c0faec80f46bfb5f827bb8ed9c384f973b
Sep 17 19:12:31.753026 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.752994    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cgwwt" event={"ID":"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd","Type":"ContainerStarted","Data":"44ccb2848d4eebc1ef5d222c6a4d79c0faec80f46bfb5f827bb8ed9c384f973b"}
Sep 17 19:12:31.754188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:31.754165    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-nhfvm" event={"ID":"032610a4-c869-474a-beff-9748c519be5f","Type":"ContainerStarted","Data":"7e65264049c91061f5b8825b7342468d846305625cc48f7253a0d79194ba3ab3"}
Sep 17 19:12:32.223791 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:32.223754    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:32.283879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:32.283844    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:12:35.777206 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:35.777167    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-7b966d7984-cgwwt" event={"ID":"05e5e562-7740-49e7-acdf-6e4bbfd6a9fd","Type":"ContainerStarted","Data":"82cd7c14c1f5d21d3bf9b366b704c842bc5f36778a694ea012ccb16ab2bac560"}
Sep 17 19:12:35.777698 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:35.777292    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:35.778700 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:35.778677    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-nhfvm" event={"ID":"032610a4-c869-474a-beff-9748c519be5f","Type":"ContainerStarted","Data":"871699c62760bb0cfe4ce443d0d4c375fc5e7ab3bc9d4994f5f3026ff07a6fb6"}
Sep 17 19:12:35.778807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:35.778792    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:12:35.801653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:35.801592    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-7b966d7984-cgwwt" podStartSLOduration=2.186784032 podStartE2EDuration="5.801579263s" totalImagesPullingTime="3.614795231s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:35.800538743 +0000 UTC m=+400.966280039" watchObservedRunningTime="2026-09-17 19:12:35.801579263 +0000 UTC m=+400.967320538"
Sep 17 19:12:36.746808 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:36.746769    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d69ffd86f-92sbw" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerName="console" containerID="cri-o://cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1" gracePeriod=15
Sep 17 19:12:36.749614 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:36.749586    2586 patch_prober.go:28] interesting pod/console-5d69ffd86f-92sbw container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection reset by peer" start-of-body=
Sep 17 19:12:36.749822 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:36.749656    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-5d69ffd86f-92sbw" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerName="console" probeResult="failure" output="Get \"https://10.133.0.17:8443/health\": dial tcp 10.133.0.17:8443: connect: connection reset by peer"
Sep 17 19:12:36.980235 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:36.980212    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:12:37.005593 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.005506    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-nhfvm" podStartSLOduration=3.408227955 podStartE2EDuration="7.005490528s" totalImagesPullingTime="3.597262573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:12:35.85049734 +0000 UTC m=+401.016238615" watchObservedRunningTime="2026-09-17 19:12:37.005490528 +0000 UTC m=+402.171231805"
Sep 17 19:12:37.084222 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084193    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084230    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jxbm\" (UniqueName: \"kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084273    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084303    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084347    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084645 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084485    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084645 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084536    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert\") pod \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\" (UID: \"a305c7ed-7fbe-4d40-b8b6-17055c72fb57\") "
Sep 17 19:12:37.084758 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084675    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:37.084888 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084829    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config" (OuterVolumeSpecName: "console-config") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:37.084959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084927    2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-trusted-ca-bundle\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.084994 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.084939    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca" (OuterVolumeSpecName: "service-ca") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:37.085186 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.085122    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:12:37.086610 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.086581    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:37.086788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.086636    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:12:37.086788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.086673    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm" (OuterVolumeSpecName: "kube-api-access-5jxbm") pod "a305c7ed-7fbe-4d40-b8b6-17055c72fb57" (UID: "a305c7ed-7fbe-4d40-b8b6-17055c72fb57"). InnerVolumeSpecName "kube-api-access-5jxbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:12:37.185660 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185635    2586 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.185660 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185656    2586 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-oauth-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.185660 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185665    2586 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-console-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.185849 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185674    2586 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-service-ca\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.185849 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185681    2586 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-oauth-serving-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.185849 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.185689    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jxbm\" (UniqueName: \"kubernetes.io/projected/a305c7ed-7fbe-4d40-b8b6-17055c72fb57-kube-api-access-5jxbm\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:12:37.788639 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.788585    2586 generic.go:356] "Generic (PLEG): container finished" podID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerID="cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1" exitCode=0
Sep 17 19:12:37.788639 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.788644    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69ffd86f-92sbw" event={"ID":"a305c7ed-7fbe-4d40-b8b6-17055c72fb57","Type":"ContainerDied","Data":"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"}
Sep 17 19:12:37.788874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.788652    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d69ffd86f-92sbw"
Sep 17 19:12:37.788874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.788682    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d69ffd86f-92sbw" event={"ID":"a305c7ed-7fbe-4d40-b8b6-17055c72fb57","Type":"ContainerDied","Data":"f699b0b5c3abb1087dacc885c8672d6d15553451a67eb70a16dd66d1d6798ef8"}
Sep 17 19:12:37.788874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.788702    2586 scope.go:117] "RemoveContainer" containerID="cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"
Sep 17 19:12:37.797555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.797537    2586 scope.go:117] "RemoveContainer" containerID="cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"
Sep 17 19:12:37.797820 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:12:37.797802    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1\": container with ID starting with cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1 not found: ID does not exist" containerID="cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"
Sep 17 19:12:37.797868 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.797828    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1"} err="failed to get container status \"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1\": rpc error: code = NotFound desc = could not find container \"cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1\": container with ID starting with cc697c00160c583c952e8dd54affc501412d1adb520b72782783fdc6c95816e1 not found: ID does not exist"
Sep 17 19:12:37.809821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.809779    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:12:37.811649 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:37.811630    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d69ffd86f-92sbw"]
Sep 17 19:12:39.016602 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:39.016568    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" path="/var/lib/kubelet/pods/a305c7ed-7fbe-4d40-b8b6-17055c72fb57/volumes"
Sep 17 19:12:46.785429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:46.785401    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-7b966d7984-cgwwt"
Sep 17 19:12:46.787436 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:12:46.787417    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-nhfvm"
Sep 17 19:13:23.009716 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:13:23.009574    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:06.745114 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.745073    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:06.760757 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.760722    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:06.761078 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.761062    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerName="console"
Sep 17 19:14:06.761153 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.761081    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerName="console"
Sep 17 19:14:06.761205 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.761152    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a305c7ed-7fbe-4d40-b8b6-17055c72fb57" containerName="console"
Sep 17 19:14:06.763898 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.763880    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.766107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.766080    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 19:14:06.766223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.766165    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4btc\""
Sep 17 19:14:06.772810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.772791    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:06.817021 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.816997    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.817122 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.817024    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j4qr\" (UniqueName: \"kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.917676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.917645    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.917676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.917680    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j4qr\" (UniqueName: \"kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.918017 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.917999    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:06.926566 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:06.926548    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j4qr\" (UniqueName: \"kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr\") pod \"seaweedfs-tls-custom-6644bf5d6c-ztnhm\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") " pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:07.073039 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:07.073010    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:07.347264 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:07.347077    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:07.349812 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:14:07.349777    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da0f7a4_bfa0_4347_b92c_f7b9cb1e9236.slice/crio-e137b325f05cb5875de125eadecc35b981e16a1103bce92644c4c2f70205a18d WatchSource:0}: Error finding container e137b325f05cb5875de125eadecc35b981e16a1103bce92644c4c2f70205a18d: Status 404 returned error can't find the container with id e137b325f05cb5875de125eadecc35b981e16a1103bce92644c4c2f70205a18d
Sep 17 19:14:08.146548 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:08.146506    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" event={"ID":"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236","Type":"ContainerStarted","Data":"e137b325f05cb5875de125eadecc35b981e16a1103bce92644c4c2f70205a18d"}
Sep 17 19:14:11.165281 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:11.165249    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" event={"ID":"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236","Type":"ContainerStarted","Data":"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"}
Sep 17 19:14:11.182823 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:11.182775    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" podStartSLOduration=2.36296359 podStartE2EDuration="5.182757368s" totalImagesPullingTime="2.819793778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:11.181523327 +0000 UTC m=+496.347264601" watchObservedRunningTime="2026-09-17 19:14:11.182757368 +0000 UTC m=+496.348498644"
Sep 17 19:14:12.434352 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:12.434310    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:13.173489 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:13.173443    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" podUID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" containerName="seaweedfs-tls-custom" containerID="cri-o://3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926" gracePeriod=30
Sep 17 19:14:40.803789 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.803762    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:40.892972 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.892939    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data\") pod \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") "
Sep 17 19:14:40.893114 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.892980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j4qr\" (UniqueName: \"kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr\") pod \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\" (UID: \"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236\") "
Sep 17 19:14:40.899501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.895003    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data" (OuterVolumeSpecName: "data") pod "0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" (UID: "0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:14:40.899501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.896108    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr" (OuterVolumeSpecName: "kube-api-access-6j4qr") pod "0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" (UID: "0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236"). InnerVolumeSpecName "kube-api-access-6j4qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:14:40.994712 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.994616    2586 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-data\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:14:40.994712 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:40.994659    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j4qr\" (UniqueName: \"kubernetes.io/projected/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236-kube-api-access-6j4qr\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:14:41.286433 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.286404    2586 generic.go:356] "Generic (PLEG): container finished" podID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" containerID="3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926" exitCode=0
Sep 17 19:14:41.286616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.286471    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"
Sep 17 19:14:41.286616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.286492    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" event={"ID":"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236","Type":"ContainerDied","Data":"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"}
Sep 17 19:14:41.286616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.286542    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm" event={"ID":"0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236","Type":"ContainerDied","Data":"e137b325f05cb5875de125eadecc35b981e16a1103bce92644c4c2f70205a18d"}
Sep 17 19:14:41.286616 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.286562    2586 scope.go:117] "RemoveContainer" containerID="3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"
Sep 17 19:14:41.295729 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.295716    2586 scope.go:117] "RemoveContainer" containerID="3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"
Sep 17 19:14:41.295992 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:14:41.295976    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926\": container with ID starting with 3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926 not found: ID does not exist" containerID="3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"
Sep 17 19:14:41.296053 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.295999    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926"} err="failed to get container status \"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926\": rpc error: code = NotFound desc = could not find container \"3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926\": container with ID starting with 3daf721b15f11caa11e6510142ed80e0b4077a5cb7ceb9b1881281fe3ebe0926 not found: ID does not exist"
Sep 17 19:14:41.301910 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.301890    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:41.304026 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.304006    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/seaweedfs-tls-custom-6644bf5d6c-ztnhm"]
Sep 17 19:14:41.336306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.336282    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"]
Sep 17 19:14:41.336637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.336606    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" containerName="seaweedfs-tls-custom"
Sep 17 19:14:41.336682 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.336638    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" containerName="seaweedfs-tls-custom"
Sep 17 19:14:41.336720 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.336691    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" containerName="seaweedfs-tls-custom"
Sep 17 19:14:41.340807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.340794    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.343029 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.343012    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Sep 17 19:14:41.343128 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.343109    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom\""
Sep 17 19:14:41.343215 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.343200    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-c4btc\""
Sep 17 19:14:41.347075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.347055    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"]
Sep 17 19:14:41.498189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.498164    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.498341 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.498207    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c7d402-e1ce-4396-9f26-6109635bbf20-data\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.498341 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.498237    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bg2\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-kube-api-access-58bg2\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.598997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.598927    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.598997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.598960    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c7d402-e1ce-4396-9f26-6109635bbf20-data\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.598997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.598981    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bg2\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-kube-api-access-58bg2\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.599321 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.599302    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/29c7d402-e1ce-4396-9f26-6109635bbf20-data\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.601509 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.601486    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-custom\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-seaweedfs-tls-custom\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.607947 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.607928    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bg2\" (UniqueName: \"kubernetes.io/projected/29c7d402-e1ce-4396-9f26-6109635bbf20-kube-api-access-58bg2\") pod \"seaweedfs-tls-custom-8b44d767d-d4lpg\" (UID: \"29c7d402-e1ce-4396-9f26-6109635bbf20\") " pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.650479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.650459    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"
Sep 17 19:14:41.917379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:41.917193    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-custom-8b44d767d-d4lpg"]
Sep 17 19:14:41.919596 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:14:41.919567    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c7d402_e1ce_4396_9f26_6109635bbf20.slice/crio-f43b629325fd0a1555632474cb32ea8ffa26236d77b3ecc9880e8554bc3a43d3 WatchSource:0}: Error finding container f43b629325fd0a1555632474cb32ea8ffa26236d77b3ecc9880e8554bc3a43d3: Status 404 returned error can't find the container with id f43b629325fd0a1555632474cb32ea8ffa26236d77b3ecc9880e8554bc3a43d3
Sep 17 19:14:42.292025 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:42.291999    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg" event={"ID":"29c7d402-e1ce-4396-9f26-6109635bbf20","Type":"ContainerStarted","Data":"a1766a8afadfb4fae459fe8d8561fef525812f36a7742a6754d5538e120e63d7"}
Sep 17 19:14:42.292239 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:42.292048    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg" event={"ID":"29c7d402-e1ce-4396-9f26-6109635bbf20","Type":"ContainerStarted","Data":"f43b629325fd0a1555632474cb32ea8ffa26236d77b3ecc9880e8554bc3a43d3"}
Sep 17 19:14:42.309430 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:42.309372    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-custom-8b44d767d-d4lpg" podStartSLOduration=1.003818146 podStartE2EDuration="1.309353822s" totalImagesPullingTime="305.535676ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:42.307315103 +0000 UTC m=+527.473056378" watchObservedRunningTime="2026-09-17 19:14:42.309353822 +0000 UTC m=+527.475095100"
Sep 17 19:14:43.014956 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:43.014926    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236" path="/var/lib/kubelet/pods/0da0f7a4-bfa0-4347-b92c-f7b9cb1e9236/volumes"
Sep 17 19:14:55.771872 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.771840    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:55.772228 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.772204    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"]
Sep 17 19:14:55.775460 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.775445    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:55.777988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.777973    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Sep 17 19:14:55.778065 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.778005    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Sep 17 19:14:55.784761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.784741    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"]
Sep 17 19:14:55.896589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.896561    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:14:55.914760 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.914738    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/719d144c-2374-4158-b31e-e24c5291ce49-data\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:55.914827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.914768    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzsrv\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-kube-api-access-zzsrv\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:55.914827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:55.914803    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.015533 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.015503    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzsrv\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-kube-api-access-zzsrv\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.015669 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.015546    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.015669 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.015595    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/719d144c-2374-4158-b31e-e24c5291ce49-data\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.015907 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.015892    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/719d144c-2374-4158-b31e-e24c5291ce49-data\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.018042 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.018016    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.023982 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.023923    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzsrv\" (UniqueName: \"kubernetes.io/projected/719d144c-2374-4158-b31e-e24c5291ce49-kube-api-access-zzsrv\") pod \"seaweedfs-tls-serving-67b7477c6d-d4kc5\" (UID: \"719d144c-2374-4158-b31e-e24c5291ce49\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.083888 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.083863    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"
Sep 17 19:14:56.350192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:56.350162    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5"]
Sep 17 19:14:56.353114 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:14:56.353090    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod719d144c_2374_4158_b31e_e24c5291ce49.slice/crio-8c71cc169adf36de11c32b2b84873918f52925016f85a12e3e4aad542fdc98c7 WatchSource:0}: Error finding container 8c71cc169adf36de11c32b2b84873918f52925016f85a12e3e4aad542fdc98c7: Status 404 returned error can't find the container with id 8c71cc169adf36de11c32b2b84873918f52925016f85a12e3e4aad542fdc98c7
Sep 17 19:14:57.355413 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:57.355380    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5" event={"ID":"719d144c-2374-4158-b31e-e24c5291ce49","Type":"ContainerStarted","Data":"e692219eaf97c87c7298590261fde3c7bde85491072ccd970cdad210fd16a214"}
Sep 17 19:14:57.355789 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:57.355431    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5" event={"ID":"719d144c-2374-4158-b31e-e24c5291ce49","Type":"ContainerStarted","Data":"8c71cc169adf36de11c32b2b84873918f52925016f85a12e3e4aad542fdc98c7"}
Sep 17 19:14:57.370856 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:14:57.370811    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-d4kc5" podStartSLOduration=2.130182156 podStartE2EDuration="2.370797081s" totalImagesPullingTime="240.614925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:14:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:14:57.369221445 +0000 UTC m=+542.534962733" watchObservedRunningTime="2026-09-17 19:14:57.370797081 +0000 UTC m=+542.536538355"
Sep 17 19:15:09.660505 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:09.660460    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:09.760529 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:09.760496    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:21.885655 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:21.885604    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:22.095771 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:22.095739    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:22.167674 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:22.167592    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:22.523468 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:22.523362    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:31.847953 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:31.847922    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:36.967120 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:36.967079    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:37.199878 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:37.199842    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:43.897361 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:43.897326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:15:56.950795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:56.950767    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:15:56.951353 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:15:56.951035    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:16:00.275893 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:16:00.275857    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:16:57.359493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:16:57.359454    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:11.378703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:11.378672    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:17.436969 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:17.436937    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:17.538138 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:17.538091    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:17.773864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:17.773799    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:37.564559 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:37.564484    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:42.713801 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:42.713765    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:17:42.915755 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:17:42.915718    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:03.086883 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:03.086845    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:03.191885 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:03.191838    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:03.653426 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:03.653397    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:13.282597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:13.282517    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:18.428979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:18.428944    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:18.785288 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:18.785253    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:23.214727 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:23.214692    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:19:23.270306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:19:23.270248    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:49.006805 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:20:49.006729    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:54.103584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:20:54.103553    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:54.513347 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:20:54.513265    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:20:56.981174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:20:56.981144    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:20:56.982284 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:20:56.982252    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:21:02.877742 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:21:02.877703    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:21:04.306413 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:21:04.306388    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:34.275244 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:34.275126    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:35.970950 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:35.970913    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:39.380511 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:39.380474    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:39.747113 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:39.747020    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:49.239306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:49.239272    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:49.297091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:49.297065    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:22:53.427115 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:22:53.427080    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:49.404634 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:23:49.404539    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:54.571105 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:23:54.571075    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:23:54.761170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:23:54.761141    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:49.528492 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:24:49.528452    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:54.683501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:24:54.683463    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:24:54.969178 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:24:54.969105    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:51.174934 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:25:51.174901    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.319739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:25:56.319696    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:56.742827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:25:56.742753    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:25:57.004537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:25:57.004461    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:25:57.006784 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:25:57.006765    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:26:35.589884 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:26:35.589850    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:40.742938 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:26:40.742910    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:41.192904 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:26:41.192871    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:26:42.792260 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:26:42.792224    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:14.260544 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:27:14.260515    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:14.351408 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:27:14.351371    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:27:54.745040 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:27:54.745007    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:11.108480 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:28:11.108400    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:13.324507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:28:13.324466    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:18.481449 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:28:18.481419    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:28:18.918703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:28:18.918671    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:15.000568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:29:15.000533    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:20.143502 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:29:20.143461    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:29:20.262117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:29:20.262076    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:16.606461 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:30:16.606423    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:22.732342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:30:22.732302    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:22.831440 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:30:22.831409    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:30:57.037463 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:30:57.037431    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:30:57.038144 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:30:57.038127    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:32:04.515180 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:04.515152    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:09.633689 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:09.633655    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:09.704184 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:09.704156    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:20.342087 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:20.342052    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:25.459425 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:25.459359    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:25.581450 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:25.581418    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:45.047599 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:45.047523    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:50.199843 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.199806    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:50.265805 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.265772    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:50.308556 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.308524    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:32:50.312008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.311987    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.314510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.314490    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:32:50.316133 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.316113    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:32:50.316223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.316161    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Sep 17 19:32:50.316223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.316118    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:32:50.316292 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.316118    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:32:50.317111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.317091    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:32:50.317215 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.317119    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Sep 17 19:32:50.323283 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.323265    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:32:50.328152 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.328128    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:50.360124 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.360098    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j666p\" (UniqueName: \"kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.360236 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.360130    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.360236 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.360166    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.360322 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.360241    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.360322 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.360281    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.414315 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.414282    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:50.461387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.461387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461372    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.461570 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461432    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j666p\" (UniqueName: \"kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.461570 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461461    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.461570 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461500    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.461855 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.461828    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.462185 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.462162    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.462558 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.462536    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.464083 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.464067    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.473470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.473452    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j666p\" (UniqueName: \"kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p\") pod \"isvc-canary-create-predictor-7c7f45c769-p8krl\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.623316 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.623282    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:32:50.899488 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.899425    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:32:50.901688 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:32:50.901660    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0568d020_d54d_459f_8d36_34ec55229505.slice/crio-11fb1eef70b7deaff3d236b3e050c95867cf036068edfd5d77ff3727f9d9b3fa WatchSource:0}: Error finding container 11fb1eef70b7deaff3d236b3e050c95867cf036068edfd5d77ff3727f9d9b3fa: Status 404 returned error can't find the container with id 11fb1eef70b7deaff3d236b3e050c95867cf036068edfd5d77ff3727f9d9b3fa
Sep 17 19:32:50.903584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:50.903566    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:32:51.758261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:51.758230    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerStarted","Data":"11fb1eef70b7deaff3d236b3e050c95867cf036068edfd5d77ff3727f9d9b3fa"}
Sep 17 19:32:54.773003 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:54.772970    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerStarted","Data":"33379d1d352ccef9df00ffe4701cf6b36da80526d10f37e1609653247e2a98ec"}
Sep 17 19:32:57.752813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:57.752784    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:32:58.791123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:58.791097    2586 generic.go:356] "Generic (PLEG): container finished" podID="0568d020-d54d-459f-8d36-34ec55229505" containerID="33379d1d352ccef9df00ffe4701cf6b36da80526d10f37e1609653247e2a98ec" exitCode=0
Sep 17 19:32:58.791510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:58.791177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerDied","Data":"33379d1d352ccef9df00ffe4701cf6b36da80526d10f37e1609653247e2a98ec"}
Sep 17 19:32:59.212360 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:32:59.212281    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:02.873015 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:02.872915    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:03.168041 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:03.167967    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:33:13.881469 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:13.881436    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerStarted","Data":"4a74d983df48d30c815c7f3a56a312447bc6f070728b0d72f7abf2b72233b61d"}
Sep 17 19:33:16.899379 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:16.899347    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerStarted","Data":"00cbaa88dad385f6be84b3e0d0b5e4d864e23daec490fdb14e1f88075498fe04"}
Sep 17 19:33:16.899772 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:16.899459    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:33:16.920647 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:16.920565    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podStartSLOduration=2.234207846 podStartE2EDuration="26.920552521s" totalImagesPullingTime="20.816105343s" totalInitContainerRuntime="3.870239332s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:32:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:33:16.91974795 +0000 UTC m=+1642.085489246" watchObservedRunningTime="2026-09-17 19:33:16.920552521 +0000 UTC m=+1642.086293795"
Sep 17 19:33:17.903563 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:17.903529    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:33:17.904746 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:17.904719    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:33:18.907446 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:18.907407    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:33:23.912037 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:23.912011    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:33:23.912499 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:23.912473    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:33:33.913346 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:33.913294    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:33:43.913041 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:43.913003    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:33:53.913156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:33:53.913113    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:34:03.026136 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:03.026103    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:03.913305 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:03.913265    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:34:08.166401 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.166309    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:08.209271 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.209246    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:34:08.237326 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.237300    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:34:08.237486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.237455    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.239834 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.239817    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-predictor-serving-cert\""
Sep 17 19:34:08.240039 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.240024    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:34:08.351056 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.351030    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:08.387383 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.387354    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.387483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.387397    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.387483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.387420    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.387553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.387492    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf72k\" (UniqueName: \"kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.387553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.387544    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.488821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.488758    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.488821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.488810    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.488984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.488846    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.488984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.488874    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.488984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.488903    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf72k\" (UniqueName: \"kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.489531 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.489511    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.489593 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.489579    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.489647 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.489593    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.491454 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.491437    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.497520 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.497503    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf72k\" (UniqueName: \"kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k\") pod \"isvc-sklearn-v2-predictor-75bb9f496d-p2m8t\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") " pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.548458 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.548432    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:08.820052 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:08.820028    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:34:08.822194 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:34:08.822166    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod155c3cab_306d_4cb2_90a3_597d444425fb.slice/crio-27ddb1a82dedd6929e5cf18bb4015e734c5e5d7b513a9feb315814a0f36f0b0d WatchSource:0}: Error finding container 27ddb1a82dedd6929e5cf18bb4015e734c5e5d7b513a9feb315814a0f36f0b0d: Status 404 returned error can't find the container with id 27ddb1a82dedd6929e5cf18bb4015e734c5e5d7b513a9feb315814a0f36f0b0d
Sep 17 19:34:09.114267 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:09.114172    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerStarted","Data":"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"}
Sep 17 19:34:09.114267 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:09.114244    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerStarted","Data":"27ddb1a82dedd6929e5cf18bb4015e734c5e5d7b513a9feb315814a0f36f0b0d"}
Sep 17 19:34:13.133536 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:13.133501    2586 generic.go:356] "Generic (PLEG): container finished" podID="155c3cab-306d-4cb2-90a3-597d444425fb" containerID="ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104" exitCode=0
Sep 17 19:34:13.133942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:13.133571    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerDied","Data":"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"}
Sep 17 19:34:13.913273 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:13.913234    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:34:14.140064 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:14.140030    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerStarted","Data":"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"}
Sep 17 19:34:14.140423 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:14.140091    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerStarted","Data":"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"}
Sep 17 19:34:14.140423 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:14.140342    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:14.159290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:14.159242    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podStartSLOduration=2.35274943 podStartE2EDuration="6.159227272s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.806477842s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:14.157508322 +0000 UTC m=+1699.323249617" watchObservedRunningTime="2026-09-17 19:34:14.159227272 +0000 UTC m=+1699.324968544"
Sep 17 19:34:15.144678 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:15.144648    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:15.145866 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:15.145842    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:16.148832 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:16.148797    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:20.164577 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:20.164549    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:34:20.165019 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:20.164821    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" containerID="cri-o://4a74d983df48d30c815c7f3a56a312447bc6f070728b0d72f7abf2b72233b61d" gracePeriod=30
Sep 17 19:34:20.165019 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:20.164912    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" containerID="cri-o://00cbaa88dad385f6be84b3e0d0b5e4d864e23daec490fdb14e1f88075498fe04" gracePeriod=30
Sep 17 19:34:20.165200 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:20.165171    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Sep 17 19:34:20.947793 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:20.947757    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:21.152768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:21.152741    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:34:21.153262 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:21.153225    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:21.172537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:21.172516    2586 generic.go:356] "Generic (PLEG): container finished" podID="0568d020-d54d-459f-8d36-34ec55229505" containerID="00cbaa88dad385f6be84b3e0d0b5e4d864e23daec490fdb14e1f88075498fe04" exitCode=2
Sep 17 19:34:21.172537 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:21.172543    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerDied","Data":"00cbaa88dad385f6be84b3e0d0b5e4d864e23daec490fdb14e1f88075498fe04"}
Sep 17 19:34:23.907957 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:23.907919    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:28.908410 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:28.908360    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:30.166611 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:30.166580    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:34:30.286209 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:30.286177    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:30.338176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:30.338142    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:31.153170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:31.153135    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:33.908588 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:33.908540    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:33.908988 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:33.908735    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:34:35.435390 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.435351    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.507795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.507754    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.571037 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.571008    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:34:35.574649 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.574633    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.577540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.577520    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Sep 17 19:34:35.577668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.577542    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Sep 17 19:34:35.586458 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.586439    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:34:35.589496 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.589470    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.607378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.607346    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.607507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.607475    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.607589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.607518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.607589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.607569    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qltzv\" (UniqueName: \"kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.607774 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.607649    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.683123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.683094    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:34:35.708285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708222    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.708285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708263    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qltzv\" (UniqueName: \"kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.708285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708283    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.708486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708339    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.708486 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708384    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.708783 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708760    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.709001 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708984    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.709045 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.708994    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.710951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.710929    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.717534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.717516    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qltzv\" (UniqueName: \"kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv\") pod \"isvc-canary-promote-predictor-78b4c78fb9-zwnbc\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:35.885334 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:35.885299    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:36.156259 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:36.156232    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:34:36.158053 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:34:36.158027    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2de1a6_ad14_4424_9f50_8a6abe7ba64e.slice/crio-722d977680969ea405b15fe6397ee2dffb3ef3eb8c77dd4f832afe2402c43b03 WatchSource:0}: Error finding container 722d977680969ea405b15fe6397ee2dffb3ef3eb8c77dd4f832afe2402c43b03: Status 404 returned error can't find the container with id 722d977680969ea405b15fe6397ee2dffb3ef3eb8c77dd4f832afe2402c43b03
Sep 17 19:34:36.243447 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:36.243413    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerStarted","Data":"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"}
Sep 17 19:34:36.243561 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:36.243471    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerStarted","Data":"722d977680969ea405b15fe6397ee2dffb3ef3eb8c77dd4f832afe2402c43b03"}
Sep 17 19:34:38.908634 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:38.908586    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:41.153557 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:41.153515    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:41.268189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:41.268156    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerID="aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03" exitCode=0
Sep 17 19:34:41.268341 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:41.268229    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerDied","Data":"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"}
Sep 17 19:34:42.274149 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.274116    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerStarted","Data":"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"}
Sep 17 19:34:42.274564 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.274176    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerStarted","Data":"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"}
Sep 17 19:34:42.274564 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.274457    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:42.274669 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.274648    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:42.275845 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.275821    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:34:42.296810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:42.296767    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podStartSLOduration=3.260146541 podStartE2EDuration="7.296754221s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.03660768s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:34:42.292513975 +0000 UTC m=+1727.458255250" watchObservedRunningTime="2026-09-17 19:34:42.296754221 +0000 UTC m=+1727.462495496"
Sep 17 19:34:43.278194 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:43.278154    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:34:43.907900 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:43.907863    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:48.284181 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:48.284153    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:34:48.284700 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:48.284673    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:34:48.908163 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:48.908121    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Sep 17 19:34:50.316421 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.316389    2586 generic.go:356] "Generic (PLEG): container finished" podID="0568d020-d54d-459f-8d36-34ec55229505" containerID="4a74d983df48d30c815c7f3a56a312447bc6f070728b0d72f7abf2b72233b61d" exitCode=137
Sep 17 19:34:50.316763 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.316428    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerDied","Data":"4a74d983df48d30c815c7f3a56a312447bc6f070728b0d72f7abf2b72233b61d"}
Sep 17 19:34:50.352779 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.352753    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:34:50.425192 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425120    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls\") pod \"0568d020-d54d-459f-8d36-34ec55229505\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") "
Sep 17 19:34:50.425336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425212    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j666p\" (UniqueName: \"kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p\") pod \"0568d020-d54d-459f-8d36-34ec55229505\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") "
Sep 17 19:34:50.425336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425242    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config\") pod \"0568d020-d54d-459f-8d36-34ec55229505\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") "
Sep 17 19:34:50.425336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425301    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert\") pod \"0568d020-d54d-459f-8d36-34ec55229505\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") "
Sep 17 19:34:50.425487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425340    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location\") pod \"0568d020-d54d-459f-8d36-34ec55229505\" (UID: \"0568d020-d54d-459f-8d36-34ec55229505\") "
Sep 17 19:34:50.425680 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425652    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0568d020-d54d-459f-8d36-34ec55229505" (UID: "0568d020-d54d-459f-8d36-34ec55229505"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:50.425799 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425692    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0568d020-d54d-459f-8d36-34ec55229505" (UID: "0568d020-d54d-459f-8d36-34ec55229505"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:34:50.425799 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.425739    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0568d020-d54d-459f-8d36-34ec55229505" (UID: "0568d020-d54d-459f-8d36-34ec55229505"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:34:50.427438 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.427416    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0568d020-d54d-459f-8d36-34ec55229505" (UID: "0568d020-d54d-459f-8d36-34ec55229505"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:34:50.427573 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.427557    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p" (OuterVolumeSpecName: "kube-api-access-j666p") pod "0568d020-d54d-459f-8d36-34ec55229505" (UID: "0568d020-d54d-459f-8d36-34ec55229505"). InnerVolumeSpecName "kube-api-access-j666p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:34:50.526448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.526415    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j666p\" (UniqueName: \"kubernetes.io/projected/0568d020-d54d-459f-8d36-34ec55229505-kube-api-access-j666p\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:34:50.526448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.526442    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:34:50.526448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.526451    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0568d020-d54d-459f-8d36-34ec55229505-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:34:50.526678 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.526461    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0568d020-d54d-459f-8d36-34ec55229505-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:34:50.526678 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:50.526469    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0568d020-d54d-459f-8d36-34ec55229505-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:34:51.153583 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.153542    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:34:51.323103 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.323072    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl" event={"ID":"0568d020-d54d-459f-8d36-34ec55229505","Type":"ContainerDied","Data":"11fb1eef70b7deaff3d236b3e050c95867cf036068edfd5d77ff3727f9d9b3fa"}
Sep 17 19:34:51.323507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.323134    2586 scope.go:117] "RemoveContainer" containerID="00cbaa88dad385f6be84b3e0d0b5e4d864e23daec490fdb14e1f88075498fe04"
Sep 17 19:34:51.323507 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.323082    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"
Sep 17 19:34:51.331358 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.331338    2586 scope.go:117] "RemoveContainer" containerID="4a74d983df48d30c815c7f3a56a312447bc6f070728b0d72f7abf2b72233b61d"
Sep 17 19:34:51.338613 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.338597    2586 scope.go:117] "RemoveContainer" containerID="33379d1d352ccef9df00ffe4701cf6b36da80526d10f37e1609653247e2a98ec"
Sep 17 19:34:51.341546 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.341525    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:34:51.344887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:51.344869    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-7c7f45c769-p8krl"]
Sep 17 19:34:53.015147 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:53.015104    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0568d020-d54d-459f-8d36-34ec55229505" path="/var/lib/kubelet/pods/0568d020-d54d-459f-8d36-34ec55229505/volumes"
Sep 17 19:34:58.285354 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:34:58.285308    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:01.153305 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:01.153267    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:35:08.285457 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:08.285416    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:11.153807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:11.153770    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:35:18.285044 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:18.285004    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:21.153942 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:21.153906    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:35:28.285465 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:28.285421    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:31.158062 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:31.154424    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:35:38.284879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:38.284785    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:38.902474 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:38.902440    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:35:38.902703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:38.902679    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" containerID="cri-o://69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570" gracePeriod=30
Sep 17 19:35:38.902795 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:38.902724    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kube-rbac-proxy" containerID="cri-o://5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250" gracePeriod=30
Sep 17 19:35:38.927096 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:38.927074    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:39.539790 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:39.539758    2586 generic.go:356] "Generic (PLEG): container finished" podID="155c3cab-306d-4cb2-90a3-597d444425fb" containerID="5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250" exitCode=2
Sep 17 19:35:39.540161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:39.539829    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerDied","Data":"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"}
Sep 17 19:35:41.149519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:41.149481    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Sep 17 19:35:41.153769 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:41.153743    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Sep 17 19:35:43.446813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.446792    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:35:43.458042 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458019    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config\") pod \"155c3cab-306d-4cb2-90a3-597d444425fb\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") "
Sep 17 19:35:43.458151 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458111    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls\") pod \"155c3cab-306d-4cb2-90a3-597d444425fb\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") "
Sep 17 19:35:43.458151 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458137    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf72k\" (UniqueName: \"kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k\") pod \"155c3cab-306d-4cb2-90a3-597d444425fb\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") "
Sep 17 19:35:43.458261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458161    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location\") pod \"155c3cab-306d-4cb2-90a3-597d444425fb\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") "
Sep 17 19:35:43.458261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458213    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert\") pod \"155c3cab-306d-4cb2-90a3-597d444425fb\" (UID: \"155c3cab-306d-4cb2-90a3-597d444425fb\") "
Sep 17 19:35:43.458375 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458278    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "155c3cab-306d-4cb2-90a3-597d444425fb" (UID: "155c3cab-306d-4cb2-90a3-597d444425fb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:35:43.458428 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458400    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.458501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458482    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "155c3cab-306d-4cb2-90a3-597d444425fb" (UID: "155c3cab-306d-4cb2-90a3-597d444425fb"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:35:43.458586 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.458560    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "155c3cab-306d-4cb2-90a3-597d444425fb" (UID: "155c3cab-306d-4cb2-90a3-597d444425fb"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:35:43.460403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.460371    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "155c3cab-306d-4cb2-90a3-597d444425fb" (UID: "155c3cab-306d-4cb2-90a3-597d444425fb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:35:43.460484 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.460433    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k" (OuterVolumeSpecName: "kube-api-access-rf72k") pod "155c3cab-306d-4cb2-90a3-597d444425fb" (UID: "155c3cab-306d-4cb2-90a3-597d444425fb"). InnerVolumeSpecName "kube-api-access-rf72k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:35:43.558887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.558807    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/155c3cab-306d-4cb2-90a3-597d444425fb-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.558887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.558835    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf72k\" (UniqueName: \"kubernetes.io/projected/155c3cab-306d-4cb2-90a3-597d444425fb-kube-api-access-rf72k\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.558887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.558848    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/155c3cab-306d-4cb2-90a3-597d444425fb-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.558887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.558857    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/155c3cab-306d-4cb2-90a3-597d444425fb-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:35:43.560045 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.560018    2586 generic.go:356] "Generic (PLEG): container finished" podID="155c3cab-306d-4cb2-90a3-597d444425fb" containerID="69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570" exitCode=0
Sep 17 19:35:43.560170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.560074    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"
Sep 17 19:35:43.560170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.560103    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerDied","Data":"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"}
Sep 17 19:35:43.560170 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.560161    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t" event={"ID":"155c3cab-306d-4cb2-90a3-597d444425fb","Type":"ContainerDied","Data":"27ddb1a82dedd6929e5cf18bb4015e734c5e5d7b513a9feb315814a0f36f0b0d"}
Sep 17 19:35:43.560284 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.560186    2586 scope.go:117] "RemoveContainer" containerID="5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"
Sep 17 19:35:43.568663 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.568648    2586 scope.go:117] "RemoveContainer" containerID="69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"
Sep 17 19:35:43.576189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.576173    2586 scope.go:117] "RemoveContainer" containerID="ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"
Sep 17 19:35:43.583220 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.583206    2586 scope.go:117] "RemoveContainer" containerID="5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"
Sep 17 19:35:43.583453 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:35:43.583437    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250\": container with ID starting with 5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250 not found: ID does not exist" containerID="5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"
Sep 17 19:35:43.583498 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.583462    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250"} err="failed to get container status \"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250\": rpc error: code = NotFound desc = could not find container \"5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250\": container with ID starting with 5d4b76899f251ae2be76bce159317224ab9194d754db6c19854ba4e81b561250 not found: ID does not exist"
Sep 17 19:35:43.583498 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.583477    2586 scope.go:117] "RemoveContainer" containerID="69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"
Sep 17 19:35:43.583773 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:35:43.583747    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570\": container with ID starting with 69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570 not found: ID does not exist" containerID="69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"
Sep 17 19:35:43.583984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.583782    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570"} err="failed to get container status \"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570\": rpc error: code = NotFound desc = could not find container \"69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570\": container with ID starting with 69165ba7999a31bcb802ad00e807e1e1ba6662be87f934816aeacbb80ca5c570 not found: ID does not exist"
Sep 17 19:35:43.583984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.583803    2586 scope.go:117] "RemoveContainer" containerID="ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"
Sep 17 19:35:43.584223 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:35:43.584181    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104\": container with ID starting with ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104 not found: ID does not exist" containerID="ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"
Sep 17 19:35:43.584328 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.584216    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104"} err="failed to get container status \"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104\": rpc error: code = NotFound desc = could not find container \"ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104\": container with ID starting with ac247687d368325dd8b1d6729d1215fb630d3f743359a53136033ea2e2cff104 not found: ID does not exist"
Sep 17 19:35:43.585971 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.585952    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:35:43.589783 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:43.589765    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-v2-predictor-75bb9f496d-p2m8t"]
Sep 17 19:35:44.070436 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:44.070411    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:44.696648 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:44.694133    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:35:45.015829 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:45.015750    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" path="/var/lib/kubelet/pods/155c3cab-306d-4cb2-90a3-597d444425fb/volumes"
Sep 17 19:35:48.285258 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:48.285221    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:35:57.065289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:57.065261    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:35:57.066694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:57.066676    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:35:58.285820 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:35:58.285792    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:36:05.529057 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.529017    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:36:05.529546 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.529318    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" containerID="cri-o://2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5" gracePeriod=30
Sep 17 19:36:05.529641 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.529516    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kube-rbac-proxy" containerID="cri-o://03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01" gracePeriod=30
Sep 17 19:36:05.553869 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.553818    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:36:05.583294 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583266    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:36:05.583578 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583567    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="storage-initializer"
Sep 17 19:36:05.583632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583579    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="storage-initializer"
Sep 17 19:36:05.583632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583592    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="storage-initializer"
Sep 17 19:36:05.583632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583597    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="storage-initializer"
Sep 17 19:36:05.583632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583604    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container"
Sep 17 19:36:05.583632 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583610    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583635    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583640    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583652    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kube-rbac-proxy"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583657    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kube-rbac-proxy"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583668    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583673    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583716    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kserve-container"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583723    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kube-rbac-proxy"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583729    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="155c3cab-306d-4cb2-90a3-597d444425fb" containerName="kserve-container"
Sep 17 19:36:05.583786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.583734    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="0568d020-d54d-459f-8d36-34ec55229505" containerName="kube-rbac-proxy"
Sep 17 19:36:05.588202 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.588186    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.590549 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.590531    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Sep 17 19:36:05.598451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.598430    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:36:05.617567 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.617518    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.617750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.617565    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.617750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.617674    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.617750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.617728    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.618022 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.617782    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvpjb\" (UniqueName: \"kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.653218 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.653172    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerID="03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01" exitCode=2
Sep 17 19:36:05.653317 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.653246    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerDied","Data":"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"}
Sep 17 19:36:05.718280 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718246    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.718413 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718292    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.718479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718428    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.718534 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.718590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718535    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvpjb\" (UniqueName: \"kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.718868 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.718834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.719073 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.719057    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.719126 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.719086    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.720782 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.720764    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.726878 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.726858    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvpjb\" (UniqueName: \"kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb\") pod \"isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:05.898477 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:05.898391    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:06.174532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:06.174508    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:36:06.176685 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:36:06.176662    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abe89bb_c0ce_4bc6_b958_38e996f875e7.slice/crio-c56ba2c07c0cb32d2e72ae77a4cea13d383a1c893dfd2ecea0f2b38dcd5367d5 WatchSource:0}: Error finding container c56ba2c07c0cb32d2e72ae77a4cea13d383a1c893dfd2ecea0f2b38dcd5367d5: Status 404 returned error can't find the container with id c56ba2c07c0cb32d2e72ae77a4cea13d383a1c893dfd2ecea0f2b38dcd5367d5
Sep 17 19:36:06.659326 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:06.659290    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerStarted","Data":"993299a4f97e1fe30c6d719b789e52e0469e564a2c2154193cfd9fa0bc9b8755"}
Sep 17 19:36:06.659812 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:06.659359    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerStarted","Data":"c56ba2c07c0cb32d2e72ae77a4cea13d383a1c893dfd2ecea0f2b38dcd5367d5"}
Sep 17 19:36:08.278974 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:08.278936    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Sep 17 19:36:08.285004 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:08.284973    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Sep 17 19:36:09.301488 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:09.301456    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:36:09.989854 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:09.989828    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:36:10.053708 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053605    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config\") pod \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") "
Sep 17 19:36:10.053708 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053687    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location\") pod \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") "
Sep 17 19:36:10.053876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053725    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert\") pod \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") "
Sep 17 19:36:10.053876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053800    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls\") pod \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") "
Sep 17 19:36:10.053876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053842    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qltzv\" (UniqueName: \"kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv\") pod \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\" (UID: \"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e\") "
Sep 17 19:36:10.054021 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.054002    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" (UID: "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:36:10.054066 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.053950    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" (UID: "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:36:10.054120 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.054103    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" (UID: "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:36:10.054158 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.054126    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:36:10.054158 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.054141    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:36:10.055979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.055953    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" (UID: "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:36:10.056068 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.055967    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv" (OuterVolumeSpecName: "kube-api-access-qltzv") pod "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" (UID: "3c2de1a6-ad14-4424-9f50-8a6abe7ba64e"). InnerVolumeSpecName "kube-api-access-qltzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:36:10.155503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.155468    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:36:10.155503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.155495    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qltzv\" (UniqueName: \"kubernetes.io/projected/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-kube-api-access-qltzv\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:36:10.155503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.155504    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:36:10.677146 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.677113    2586 generic.go:356] "Generic (PLEG): container finished" podID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerID="993299a4f97e1fe30c6d719b789e52e0469e564a2c2154193cfd9fa0bc9b8755" exitCode=0
Sep 17 19:36:10.677532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.677185    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerDied","Data":"993299a4f97e1fe30c6d719b789e52e0469e564a2c2154193cfd9fa0bc9b8755"}
Sep 17 19:36:10.679277 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.679254    2586 generic.go:356] "Generic (PLEG): container finished" podID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerID="2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5" exitCode=0
Sep 17 19:36:10.679399 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.679308    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerDied","Data":"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"}
Sep 17 19:36:10.679399 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.679358    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc" event={"ID":"3c2de1a6-ad14-4424-9f50-8a6abe7ba64e","Type":"ContainerDied","Data":"722d977680969ea405b15fe6397ee2dffb3ef3eb8c77dd4f832afe2402c43b03"}
Sep 17 19:36:10.679399 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.679310    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"
Sep 17 19:36:10.679399 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.679395    2586 scope.go:117] "RemoveContainer" containerID="03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"
Sep 17 19:36:10.690107 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.690090    2586 scope.go:117] "RemoveContainer" containerID="2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"
Sep 17 19:36:10.697593 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.697576    2586 scope.go:117] "RemoveContainer" containerID="aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"
Sep 17 19:36:10.710400 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.710381    2586 scope.go:117] "RemoveContainer" containerID="03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"
Sep 17 19:36:10.710705 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:36:10.710678    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01\": container with ID starting with 03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01 not found: ID does not exist" containerID="03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"
Sep 17 19:36:10.710807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.710718    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01"} err="failed to get container status \"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01\": rpc error: code = NotFound desc = could not find container \"03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01\": container with ID starting with 03ae5d78a5ff5e9e2939ef2ff7b41f6f454f45a7ca8700597e9bd4aff6592d01 not found: ID does not exist"
Sep 17 19:36:10.710807 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.710743    2586 scope.go:117] "RemoveContainer" containerID="2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"
Sep 17 19:36:10.711070 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:36:10.711034    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5\": container with ID starting with 2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5 not found: ID does not exist" containerID="2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"
Sep 17 19:36:10.711125 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.711078    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5"} err="failed to get container status \"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5\": rpc error: code = NotFound desc = could not find container \"2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5\": container with ID starting with 2e1ac16bd6fb4af1d3aa4774f0edf765f9b8d7960a6fc5e64b7236729a5589b5 not found: ID does not exist"
Sep 17 19:36:10.711125 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.711096    2586 scope.go:117] "RemoveContainer" containerID="aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"
Sep 17 19:36:10.711372 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:36:10.711357    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03\": container with ID starting with aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03 not found: ID does not exist" containerID="aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"
Sep 17 19:36:10.711433 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.711374    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03"} err="failed to get container status \"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03\": rpc error: code = NotFound desc = could not find container \"aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03\": container with ID starting with aabeae9ec2eda4016c69f565e06c68c82d7d17fd4c723cd75719a7ba9f66db03 not found: ID does not exist"
Sep 17 19:36:10.716036 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.716015    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:36:10.718519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:10.718499    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-78b4c78fb9-zwnbc"]
Sep 17 19:36:11.016542 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:11.016468    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" path="/var/lib/kubelet/pods/3c2de1a6-ad14-4424-9f50-8a6abe7ba64e/volumes"
Sep 17 19:36:11.686056 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:11.686027    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerStarted","Data":"e6d0864cf7a8309242ae07922e2dc090fc664ef52f6915558d8946c95a2faae6"}
Sep 17 19:36:11.686445 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:11.686088    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerStarted","Data":"bba982229b4dc1d385af5c47d153a54b77eb0b7685a96b6becaae7e8317763e1"}
Sep 17 19:36:11.686445 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:11.686278    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:11.708135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:11.708085    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podStartSLOduration=2.9040573849999998 podStartE2EDuration="6.708070895s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.80401351s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:36:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:36:11.706968709 +0000 UTC m=+1816.872710001" watchObservedRunningTime="2026-09-17 19:36:11.708070895 +0000 UTC m=+1816.873812171"
Sep 17 19:36:12.690924 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:12.690894    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:12.692179 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:12.692153    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:13.694876 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:13.694830    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:18.700104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:18.700077    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:36:18.700644 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:18.700598    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:28.701535 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:28.701494    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:38.700510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:38.700468    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:48.701243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:48.701207    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:36:58.701066 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:36:58.701027    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:37:08.701296 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:08.701204    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:37:14.906293 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:14.906263    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:15.975915 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:15.975878    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:18.700543 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:18.700504    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:37:20.021597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:20.021559    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:20.155156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:20.155123    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:28.701614 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:28.701577    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:37:35.691656 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:35.691609    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:37:35.692082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:35.691886    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" containerID="cri-o://bba982229b4dc1d385af5c47d153a54b77eb0b7685a96b6becaae7e8317763e1" gracePeriod=30
Sep 17 19:37:35.692082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:35.691951    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kube-rbac-proxy" containerID="cri-o://e6d0864cf7a8309242ae07922e2dc090fc664ef52f6915558d8946c95a2faae6" gracePeriod=30
Sep 17 19:37:35.718019 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:35.717989    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:36.046358 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:36.046329    2586 generic.go:356] "Generic (PLEG): container finished" podID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerID="e6d0864cf7a8309242ae07922e2dc090fc664ef52f6915558d8946c95a2faae6" exitCode=2
Sep 17 19:37:36.046526 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:36.046385    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerDied","Data":"e6d0864cf7a8309242ae07922e2dc090fc664ef52f6915558d8946c95a2faae6"}
Sep 17 19:37:38.695197 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:38.695153    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Sep 17 19:37:38.700718 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:38.700691    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Sep 17 19:37:40.063780 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.063748    2586 generic.go:356] "Generic (PLEG): container finished" podID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerID="bba982229b4dc1d385af5c47d153a54b77eb0b7685a96b6becaae7e8317763e1" exitCode=0
Sep 17 19:37:40.064175 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.063822    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerDied","Data":"bba982229b4dc1d385af5c47d153a54b77eb0b7685a96b6becaae7e8317763e1"}
Sep 17 19:37:40.143199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.143176    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:37:40.264813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.264748    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvpjb\" (UniqueName: \"kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb\") pod \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") "
Sep 17 19:37:40.264813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.264779    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert\") pod \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") "
Sep 17 19:37:40.265012 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.264831    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location\") pod \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") "
Sep 17 19:37:40.265012 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.264886    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls\") pod \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") "
Sep 17 19:37:40.265012 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.264949    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config\") pod \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\" (UID: \"9abe89bb-c0ce-4bc6-b958-38e996f875e7\") "
Sep 17 19:37:40.265225 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.265171    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9abe89bb-c0ce-4bc6-b958-38e996f875e7" (UID: "9abe89bb-c0ce-4bc6-b958-38e996f875e7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:40.265285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.265187    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9abe89bb-c0ce-4bc6-b958-38e996f875e7" (UID: "9abe89bb-c0ce-4bc6-b958-38e996f875e7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:37:40.265352 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.265333    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9abe89bb-c0ce-4bc6-b958-38e996f875e7" (UID: "9abe89bb-c0ce-4bc6-b958-38e996f875e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:37:40.267085 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.267069    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9abe89bb-c0ce-4bc6-b958-38e996f875e7" (UID: "9abe89bb-c0ce-4bc6-b958-38e996f875e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:37:40.267141 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.267089    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb" (OuterVolumeSpecName: "kube-api-access-gvpjb") pod "9abe89bb-c0ce-4bc6-b958-38e996f875e7" (UID: "9abe89bb-c0ce-4bc6-b958-38e996f875e7"). InnerVolumeSpecName "kube-api-access-gvpjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:37:40.365824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.365790    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9abe89bb-c0ce-4bc6-b958-38e996f875e7-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.365824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.365819    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.365824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.365829    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvpjb\" (UniqueName: \"kubernetes.io/projected/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kube-api-access-gvpjb\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.366045 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.365838    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9abe89bb-c0ce-4bc6-b958-38e996f875e7-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.366045 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.365849    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9abe89bb-c0ce-4bc6-b958-38e996f875e7-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:37:40.913301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.913271    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:40.998703 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:40.998670    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:41.031454 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031429    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:37:41.031762 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031748    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031765    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031773    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="storage-initializer"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031778    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="storage-initializer"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031788    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031794    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container"
Sep 17 19:37:41.031803 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031803    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="storage-initializer"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031809    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="storage-initializer"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031815    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031820    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031825    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kube-rbac-proxy"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031831    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kube-rbac-proxy"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031876    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kserve-container"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031885    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kserve-container"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031893    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c2de1a6-ad14-4424-9f50-8a6abe7ba64e" containerName="kube-rbac-proxy"
Sep 17 19:37:41.031993 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.031899    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" containerName="kube-rbac-proxy"
Sep 17 19:37:41.035076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.035058    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.040691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.040673    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Sep 17 19:37:41.040691 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.040684    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Sep 17 19:37:41.056308 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.056281    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:37:41.071448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.071425    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"
Sep 17 19:37:41.071865 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.071440    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d" event={"ID":"9abe89bb-c0ce-4bc6-b958-38e996f875e7","Type":"ContainerDied","Data":"c56ba2c07c0cb32d2e72ae77a4cea13d383a1c893dfd2ecea0f2b38dcd5367d5"}
Sep 17 19:37:41.071865 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.071512    2586 scope.go:117] "RemoveContainer" containerID="e6d0864cf7a8309242ae07922e2dc090fc664ef52f6915558d8946c95a2faae6"
Sep 17 19:37:41.079762 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.079744    2586 scope.go:117] "RemoveContainer" containerID="bba982229b4dc1d385af5c47d153a54b77eb0b7685a96b6becaae7e8317763e1"
Sep 17 19:37:41.087522 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.087506    2586 scope.go:117] "RemoveContainer" containerID="993299a4f97e1fe30c6d719b789e52e0469e564a2c2154193cfd9fa0bc9b8755"
Sep 17 19:37:41.088499 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.088474    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:37:41.098189 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.098165    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-79d9bd5fdb-crh9d"]
Sep 17 19:37:41.173610 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.173526    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfkfj\" (UniqueName: \"kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.173610 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.173557    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.173846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.173670    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.173846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.173696    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.173846 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.173771    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.226977 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.226955    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:41.275100 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275072    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfkfj\" (UniqueName: \"kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275232 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275103    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275232 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275136    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275232 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275152    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275232 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275182    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275438 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:37:41.275282    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:37:41.275438 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:37:41.275331    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls podName:58bc7427-7f60-427f-84fb-ffe75f30ab9b nodeName:}" failed. No retries permitted until 2026-09-17 19:37:41.775316087 +0000 UTC m=+1906.941057340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls") pod "isvc-canary-rollback-predictor-97977b6b5-77zqs" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Sep 17 19:37:41.275615 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275592    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275877 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275857    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.275953 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.275928    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.286006 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.285985    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfkfj\" (UniqueName: \"kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.439694 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.439599    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:37:41.779770 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.779739    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.782288 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.782258    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") pod \"isvc-canary-rollback-predictor-97977b6b5-77zqs\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:41.944706 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:41.944671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:42.217863 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:42.217834    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:37:42.220168 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:37:42.220143    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bc7427_7f60_427f_84fb_ffe75f30ab9b.slice/crio-33d13d4ab57408bdf02ec655bf55539ffe35e7d76b6bc01088107a226aaa97f7 WatchSource:0}: Error finding container 33d13d4ab57408bdf02ec655bf55539ffe35e7d76b6bc01088107a226aaa97f7: Status 404 returned error can't find the container with id 33d13d4ab57408bdf02ec655bf55539ffe35e7d76b6bc01088107a226aaa97f7
Sep 17 19:37:43.016204 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:43.016172    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9abe89bb-c0ce-4bc6-b958-38e996f875e7" path="/var/lib/kubelet/pods/9abe89bb-c0ce-4bc6-b958-38e996f875e7/volumes"
Sep 17 19:37:43.082736 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:43.082703    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerStarted","Data":"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"}
Sep 17 19:37:43.082879 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:43.082762    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerStarted","Data":"33d13d4ab57408bdf02ec655bf55539ffe35e7d76b6bc01088107a226aaa97f7"}
Sep 17 19:37:47.100892 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:47.100859    2586 generic.go:356] "Generic (PLEG): container finished" podID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerID="7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164" exitCode=0
Sep 17 19:37:47.101267 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:47.100940    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerDied","Data":"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"}
Sep 17 19:37:48.107788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.107754    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerStarted","Data":"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"}
Sep 17 19:37:48.108211 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.107811    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerStarted","Data":"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"}
Sep 17 19:37:48.108211 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.108063    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:48.108211 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.108202    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:48.109435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.109410    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:37:48.128161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:48.128115    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podStartSLOduration=3.2219739 podStartE2EDuration="7.128101815s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.906127915s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:37:48.126683667 +0000 UTC m=+1913.292424965" watchObservedRunningTime="2026-09-17 19:37:48.128101815 +0000 UTC m=+1913.293843145"
Sep 17 19:37:49.112565 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:49.112518    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:37:50.116695 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:50.116652    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:37:55.121329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:55.121301    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:37:55.121891 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:37:55.121861    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:00.697615 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:00.697583    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:05.122199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:05.122151    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:05.860208 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:05.860171    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:06.144290 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:06.144209    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:15.121869 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:15.121832    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:25.122015 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:25.121978    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:26.683199 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:26.683162    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:35.122397 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:35.122358    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:45.122181 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:45.122098    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:38:47.503249 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:47.503213    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:52.635834 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:52.635793    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:52.765709 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:52.765648    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:38:55.122496 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:38:55.122454    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:39:05.122671 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:39:05.122644    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:39:20.965521 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:39:20.965483    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:39:44.366222 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:39:44.366145    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:44.660696 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:40:44.660601    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:44.761458 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:40:44.761428    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:40:57.090127 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:40:57.090095    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:40:57.092331 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:40:57.092308    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:41:11.128069 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.128040    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:41:11.128493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.128272    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kube-rbac-proxy" containerID="cri-o://2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a" gracePeriod=30
Sep 17 19:41:11.128493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.128253    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" containerID="cri-o://21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4" gracePeriod=30
Sep 17 19:41:11.153246 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.153223    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:11.979541 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.979506    2586 generic.go:356] "Generic (PLEG): container finished" podID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerID="2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a" exitCode=2
Sep 17 19:41:11.979732 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:11.979581    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerDied","Data":"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"}
Sep 17 19:41:15.117219 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.117177    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Sep 17 19:41:15.122155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.122118    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Sep 17 19:41:15.468788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.468766    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:41:15.598268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598225    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfkfj\" (UniqueName: \"kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj\") pod \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") "
Sep 17 19:41:15.598451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598290    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location\") pod \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") "
Sep 17 19:41:15.598451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598336    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config\") pod \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") "
Sep 17 19:41:15.598451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598396    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") pod \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") "
Sep 17 19:41:15.598588 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598453    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert\") pod \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\" (UID: \"58bc7427-7f60-427f-84fb-ffe75f30ab9b\") "
Sep 17 19:41:15.598674 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598641    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58bc7427-7f60-427f-84fb-ffe75f30ab9b" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:41:15.598746 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598724    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:41:15.598823 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598799    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58bc7427-7f60-427f-84fb-ffe75f30ab9b" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:15.598891 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.598863    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58bc7427-7f60-427f-84fb-ffe75f30ab9b" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:41:15.600608 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.600583    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58bc7427-7f60-427f-84fb-ffe75f30ab9b" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:41:15.600608 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.600601    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj" (OuterVolumeSpecName: "kube-api-access-xfkfj") pod "58bc7427-7f60-427f-84fb-ffe75f30ab9b" (UID: "58bc7427-7f60-427f-84fb-ffe75f30ab9b"). InnerVolumeSpecName "kube-api-access-xfkfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:41:15.699055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.699001    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:41:15.699055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.699020    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfkfj\" (UniqueName: \"kubernetes.io/projected/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-api-access-xfkfj\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:41:15.699055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.699029    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bc7427-7f60-427f-84fb-ffe75f30ab9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:41:15.699055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.699040    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bc7427-7f60-427f-84fb-ffe75f30ab9b-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:41:15.997676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.997575    2586 generic.go:356] "Generic (PLEG): container finished" podID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerID="21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4" exitCode=0
Sep 17 19:41:15.997676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.997651    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"
Sep 17 19:41:15.997676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.997659    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerDied","Data":"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"}
Sep 17 19:41:15.998030 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.997721    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs" event={"ID":"58bc7427-7f60-427f-84fb-ffe75f30ab9b","Type":"ContainerDied","Data":"33d13d4ab57408bdf02ec655bf55539ffe35e7d76b6bc01088107a226aaa97f7"}
Sep 17 19:41:15.998030 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:15.997747    2586 scope.go:117] "RemoveContainer" containerID="2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"
Sep 17 19:41:16.006236 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.006218    2586 scope.go:117] "RemoveContainer" containerID="21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"
Sep 17 19:41:16.013575 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.013561    2586 scope.go:117] "RemoveContainer" containerID="7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"
Sep 17 19:41:16.023659 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.023638    2586 scope.go:117] "RemoveContainer" containerID="2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"
Sep 17 19:41:16.023916 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:41:16.023895    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a\": container with ID starting with 2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a not found: ID does not exist" containerID="2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"
Sep 17 19:41:16.023979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.023943    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a"} err="failed to get container status \"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a\": rpc error: code = NotFound desc = could not find container \"2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a\": container with ID starting with 2a0d1c729b0bc269b4b6761af7e41c85d2ba6e5233b32f0df07b0a42aa55e06a not found: ID does not exist"
Sep 17 19:41:16.023979 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.023960    2586 scope.go:117] "RemoveContainer" containerID="21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"
Sep 17 19:41:16.024198 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:41:16.024181    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4\": container with ID starting with 21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4 not found: ID does not exist" containerID="21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"
Sep 17 19:41:16.024257 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.024201    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4"} err="failed to get container status \"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4\": rpc error: code = NotFound desc = could not find container \"21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4\": container with ID starting with 21e8c36cd9f132e346cdc3fec152eec4ffb073ddb372f4f06233def89259e8d4 not found: ID does not exist"
Sep 17 19:41:16.024257 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.024213    2586 scope.go:117] "RemoveContainer" containerID="7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"
Sep 17 19:41:16.024448 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:41:16.024434    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164\": container with ID starting with 7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164 not found: ID does not exist" containerID="7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"
Sep 17 19:41:16.024493 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.024450    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164"} err="failed to get container status \"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164\": rpc error: code = NotFound desc = could not find container \"7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164\": container with ID starting with 7f547719ae1d3a59d898c068c2fd5bd8cc6b51625f78fa1d29a1157081191164 not found: ID does not exist"
Sep 17 19:41:16.025035 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.025017    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:41:16.030655 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.030635    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-97977b6b5-77zqs"]
Sep 17 19:41:16.199353 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.199326    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:16.263241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.263174    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:16.300025 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.299993    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:41:16.300326 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300313    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container"
Sep 17 19:41:16.300372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300328    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container"
Sep 17 19:41:16.300372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300338    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kube-rbac-proxy"
Sep 17 19:41:16.300372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300344    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kube-rbac-proxy"
Sep 17 19:41:16.300372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300366    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="storage-initializer"
Sep 17 19:41:16.300496 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300373    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="storage-initializer"
Sep 17 19:41:16.300496 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300428    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kserve-container"
Sep 17 19:41:16.300496 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.300436    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" containerName="kube-rbac-proxy"
Sep 17 19:41:16.304841 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.304823    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.307329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307306    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:41:16.307434 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307382    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:41:16.307487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307382    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:41:16.307487 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307386    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Sep 17 19:41:16.307788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307755    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:41:16.307920 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307814    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Sep 17 19:41:16.307920 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.307846    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:41:16.316121 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.316101    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:41:16.332377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.332344    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:16.364757 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.364732    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:41:16.368060 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.368043    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.370412 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.370388    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Sep 17 19:41:16.378173 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.378155    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:41:16.404689 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.404658    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcgw\" (UniqueName: \"kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.404813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.404737    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.404813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.404770    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.404915 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.404858    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.404915 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.404882    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.500174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.500143    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:16.505222 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505200    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.505350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505250    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcgw\" (UniqueName: \"kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.505350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505282    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.505350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505313    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.505519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505363    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.505519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505399    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.505519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505429    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skdqw\" (UniqueName: \"kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.505519 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505484    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.505731 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505537    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.505731 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505645    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.505821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.505802    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.506065 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.506043    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.506166 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.506066    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.508135 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.508112    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.514972 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.514923    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcgw\" (UniqueName: \"kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw\") pod \"isvc-canary-stop-v2-predictor-d65d59944-2lf2n\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.606453 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606425    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.606584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606456    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.606584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606477    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skdqw\" (UniqueName: \"kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.606584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606524    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.606584 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606541    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.606974 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.606953    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.607161 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.607141    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.607243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.607190    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.608917 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.608889    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.614689 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.614671    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:16.615004 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.614988    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skdqw\" (UniqueName: \"kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw\") pod \"isvc-canary-stop-predictor-55656fdf77-vgcxq\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.678155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.678120    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:16.889673 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.889647    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:41:16.891114 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:41:16.891089    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39e280e1_aa16_49eb_b4f7_9c8f9af6a7fe.slice/crio-731d4edbdc459722d281c630620093d20c54729cb31925dfbb0aa1e201186ef8 WatchSource:0}: Error finding container 731d4edbdc459722d281c630620093d20c54729cb31925dfbb0aa1e201186ef8: Status 404 returned error can't find the container with id 731d4edbdc459722d281c630620093d20c54729cb31925dfbb0aa1e201186ef8
Sep 17 19:41:16.950315 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:16.950282    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:41:16.953095 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:41:16.953071    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708b0e1c_35be_4816_ae39_f67640626b9b.slice/crio-5058a69f1880bdf84e31dca3159b32a9ea64070fff4d021ffd3dc645dfbafb29 WatchSource:0}: Error finding container 5058a69f1880bdf84e31dca3159b32a9ea64070fff4d021ffd3dc645dfbafb29: Status 404 returned error can't find the container with id 5058a69f1880bdf84e31dca3159b32a9ea64070fff4d021ffd3dc645dfbafb29
Sep 17 19:41:17.003781 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:17.003744    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerStarted","Data":"5058a69f1880bdf84e31dca3159b32a9ea64070fff4d021ffd3dc645dfbafb29"}
Sep 17 19:41:17.006329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:17.006289    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerStarted","Data":"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"}
Sep 17 19:41:17.006432 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:17.006344    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerStarted","Data":"731d4edbdc459722d281c630620093d20c54729cb31925dfbb0aa1e201186ef8"}
Sep 17 19:41:17.016391 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:17.016365    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bc7427-7f60-427f-84fb-ffe75f30ab9b" path="/var/lib/kubelet/pods/58bc7427-7f60-427f-84fb-ffe75f30ab9b/volumes"
Sep 17 19:41:18.012141 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:18.012106    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerStarted","Data":"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"}
Sep 17 19:41:21.032094 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.032058    2586 generic.go:356] "Generic (PLEG): container finished" podID="708b0e1c-35be-4816-ae39-f67640626b9b" containerID="02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451" exitCode=0
Sep 17 19:41:21.032455 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.032124    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerDied","Data":"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"}
Sep 17 19:41:21.033813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.033793    2586 generic.go:356] "Generic (PLEG): container finished" podID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerID="4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4" exitCode=0
Sep 17 19:41:21.033919 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.033885    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerDied","Data":"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"}
Sep 17 19:41:21.309300 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.309270    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:21.554590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:21.554515    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:22.040396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.040363    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerStarted","Data":"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"}
Sep 17 19:41:22.040864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.040432    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerStarted","Data":"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"}
Sep 17 19:41:22.040864 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.040653    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:22.042637 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.042598    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerStarted","Data":"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"}
Sep 17 19:41:22.042739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.042673    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerStarted","Data":"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"}
Sep 17 19:41:22.042881 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.042864    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:22.058789 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.058745    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podStartSLOduration=2.074395654 podStartE2EDuration="6.058730978s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.984335324s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:22.057579611 +0000 UTC m=+2127.223320899" watchObservedRunningTime="2026-09-17 19:41:22.058730978 +0000 UTC m=+2127.224472252"
Sep 17 19:41:22.076064 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:22.076022    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podStartSLOduration=2.013903851 podStartE2EDuration="6.076010509s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.062106658s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:41:22.075253672 +0000 UTC m=+2127.240994968" watchObservedRunningTime="2026-09-17 19:41:22.076010509 +0000 UTC m=+2127.241751784"
Sep 17 19:41:23.051822 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:23.051787    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:23.052266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:23.051884    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:23.052951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:23.052923    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:23.053066 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:23.052982    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:41:24.056217 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:24.056165    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:41:24.056664 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:24.056288    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:29.060178 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:29.060154    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:41:29.060659 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:29.060257    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:41:29.060799 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:29.060773    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:29.061130 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:29.061098    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:41:35.994075 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:35.993968    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:36.095581 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:36.095550    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:41:39.061145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:39.061103    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:41:39.061145 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:39.061115    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:49.061352 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:49.061312    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:49.061850 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:49.061315    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:41:59.060848 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:59.060804    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:41:59.061276 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:41:59.061095    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:42:09.061467 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:09.061425    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:42:09.061955 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:09.061466    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:42:19.060862 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:19.060827    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:42:19.061326 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:19.061051    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:42:29.060808 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:29.060766    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:42:29.061332 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:29.061042    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:42:39.062421 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:39.062390    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:42:39.062934 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:39.062489    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:42:46.283955 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.283923    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:42:46.284470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.284189    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" containerID="cri-o://647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0" gracePeriod=30
Sep 17 19:42:46.284470 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.284226    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kube-rbac-proxy" containerID="cri-o://a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78" gracePeriod=30
Sep 17 19:42:46.291117 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.291084    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:46.318083 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.318056    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:42:46.318333 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.318302    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" containerID="cri-o://a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79" gracePeriod=30
Sep 17 19:42:46.318439 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.318312    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kube-rbac-proxy" containerID="cri-o://9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6" gracePeriod=30
Sep 17 19:42:46.423421 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.423393    2586 generic.go:356] "Generic (PLEG): container finished" podID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerID="a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78" exitCode=2
Sep 17 19:42:46.423604 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:46.423467    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerDied","Data":"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"}
Sep 17 19:42:47.297473 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:47.297438    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:47.429930 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:47.429902    2586 generic.go:356] "Generic (PLEG): container finished" podID="708b0e1c-35be-4816-ae39-f67640626b9b" containerID="9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6" exitCode=2
Sep 17 19:42:47.430091 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:47.429974    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerDied","Data":"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"}
Sep 17 19:42:49.056919 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:49.056878    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Sep 17 19:42:49.057324 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:49.056887    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Sep 17 19:42:49.061448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:49.061422    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Sep 17 19:42:49.061550 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:49.061427    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Sep 17 19:42:50.917818 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.917794    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:42:50.936577 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936557    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert\") pod \"708b0e1c-35be-4816-ae39-f67640626b9b\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") "
Sep 17 19:42:50.936714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936592    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location\") pod \"708b0e1c-35be-4816-ae39-f67640626b9b\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") "
Sep 17 19:42:50.936714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936615    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skdqw\" (UniqueName: \"kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw\") pod \"708b0e1c-35be-4816-ae39-f67640626b9b\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") "
Sep 17 19:42:50.936714 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936675    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls\") pod \"708b0e1c-35be-4816-ae39-f67640626b9b\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") "
Sep 17 19:42:50.936882 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936723    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config\") pod \"708b0e1c-35be-4816-ae39-f67640626b9b\" (UID: \"708b0e1c-35be-4816-ae39-f67640626b9b\") "
Sep 17 19:42:50.936950 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936922    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "708b0e1c-35be-4816-ae39-f67640626b9b" (UID: "708b0e1c-35be-4816-ae39-f67640626b9b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:50.937004 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.936972    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "708b0e1c-35be-4816-ae39-f67640626b9b" (UID: "708b0e1c-35be-4816-ae39-f67640626b9b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:50.937253 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.937232    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "708b0e1c-35be-4816-ae39-f67640626b9b" (UID: "708b0e1c-35be-4816-ae39-f67640626b9b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:50.939982 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.939946    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw" (OuterVolumeSpecName: "kube-api-access-skdqw") pod "708b0e1c-35be-4816-ae39-f67640626b9b" (UID: "708b0e1c-35be-4816-ae39-f67640626b9b"). InnerVolumeSpecName "kube-api-access-skdqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:42:50.940704 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.940681    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "708b0e1c-35be-4816-ae39-f67640626b9b" (UID: "708b0e1c-35be-4816-ae39-f67640626b9b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:42:50.950985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:50.950938    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:42:51.037671 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.037644    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config\") pod \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") "
Sep 17 19:42:51.037785 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.037715    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gmcgw\" (UniqueName: \"kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw\") pod \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") "
Sep 17 19:42:51.037785 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.037760    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location\") pod \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") "
Sep 17 19:42:51.037897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.037788    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert\") pod \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") "
Sep 17 19:42:51.037897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.037831    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls\") pod \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\" (UID: \"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe\") "
Sep 17 19:42:51.038047 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038029    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.038111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038059    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/708b0e1c-35be-4816-ae39-f67640626b9b-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.038111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038075    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skdqw\" (UniqueName: \"kubernetes.io/projected/708b0e1c-35be-4816-ae39-f67640626b9b-kube-api-access-skdqw\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.038111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038107    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/708b0e1c-35be-4816-ae39-f67640626b9b-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.038261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038022    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" (UID: "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:51.038261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038124    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/708b0e1c-35be-4816-ae39-f67640626b9b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.038261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038072    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" (UID: "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:42:51.038261 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.038147    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" (UID: "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:42:51.058941 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.058914    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" (UID: "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:42:51.068892 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.068871    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw" (OuterVolumeSpecName: "kube-api-access-gmcgw") pod "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" (UID: "39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe"). InnerVolumeSpecName "kube-api-access-gmcgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:42:51.139314 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.139288    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcgw\" (UniqueName: \"kubernetes.io/projected/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-api-access-gmcgw\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.139314 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.139314    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.139444 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.139328    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.139444 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.139342    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.139444 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.139353    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:42:51.449359 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.449324    2586 generic.go:356] "Generic (PLEG): container finished" podID="708b0e1c-35be-4816-ae39-f67640626b9b" containerID="a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79" exitCode=0
Sep 17 19:42:51.449555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.449373    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"
Sep 17 19:42:51.449555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.449405    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerDied","Data":"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"}
Sep 17 19:42:51.449555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.449463    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq" event={"ID":"708b0e1c-35be-4816-ae39-f67640626b9b","Type":"ContainerDied","Data":"5058a69f1880bdf84e31dca3159b32a9ea64070fff4d021ffd3dc645dfbafb29"}
Sep 17 19:42:51.449555 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.449493    2586 scope.go:117] "RemoveContainer" containerID="9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"
Sep 17 19:42:51.451871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.451847    2586 generic.go:356] "Generic (PLEG): container finished" podID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerID="647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0" exitCode=0
Sep 17 19:42:51.452013 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.451920    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"
Sep 17 19:42:51.452013 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.451923    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerDied","Data":"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"}
Sep 17 19:42:51.452134 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.452047    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n" event={"ID":"39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe","Type":"ContainerDied","Data":"731d4edbdc459722d281c630620093d20c54729cb31925dfbb0aa1e201186ef8"}
Sep 17 19:42:51.458433 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.458415    2586 scope.go:117] "RemoveContainer" containerID="a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"
Sep 17 19:42:51.466292 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.466266    2586 scope.go:117] "RemoveContainer" containerID="02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"
Sep 17 19:42:51.467892 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.467876    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:42:51.474451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.474432    2586 scope.go:117] "RemoveContainer" containerID="9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"
Sep 17 19:42:51.474750 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.474730    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6\": container with ID starting with 9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6 not found: ID does not exist" containerID="9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"
Sep 17 19:42:51.474810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.474758    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6"} err="failed to get container status \"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6\": rpc error: code = NotFound desc = could not find container \"9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6\": container with ID starting with 9c60ea191f3578957978ec170a9e16f3c48220ae005533f6b920988bb87ce0b6 not found: ID does not exist"
Sep 17 19:42:51.474810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.474780    2586 scope.go:117] "RemoveContainer" containerID="a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"
Sep 17 19:42:51.475049 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.475031    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-55656fdf77-vgcxq"]
Sep 17 19:42:51.475104 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.475042    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79\": container with ID starting with a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79 not found: ID does not exist" containerID="a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"
Sep 17 19:42:51.475104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.475063    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79"} err="failed to get container status \"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79\": rpc error: code = NotFound desc = could not find container \"a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79\": container with ID starting with a3f120fd8d3856bbb2fc2fe652484bb52acb891664cb1e0ee7e7ae5facb61d79 not found: ID does not exist"
Sep 17 19:42:51.475104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.475078    2586 scope.go:117] "RemoveContainer" containerID="02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"
Sep 17 19:42:51.475332 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.475314    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451\": container with ID starting with 02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451 not found: ID does not exist" containerID="02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"
Sep 17 19:42:51.475372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.475337    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451"} err="failed to get container status \"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451\": rpc error: code = NotFound desc = could not find container \"02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451\": container with ID starting with 02d18d14f48aa5d29ada20247a1eaa0420a31cdabf481bd84486b2d4491a2451 not found: ID does not exist"
Sep 17 19:42:51.475372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.475352    2586 scope.go:117] "RemoveContainer" containerID="a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"
Sep 17 19:42:51.482614 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.482599    2586 scope.go:117] "RemoveContainer" containerID="647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"
Sep 17 19:42:51.489553 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.489535    2586 scope.go:117] "RemoveContainer" containerID="4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"
Sep 17 19:42:51.492176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.492153    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:42:51.496196 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.496175    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-d65d59944-2lf2n"]
Sep 17 19:42:51.497648 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.497613    2586 scope.go:117] "RemoveContainer" containerID="a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"
Sep 17 19:42:51.497905 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.497888    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78\": container with ID starting with a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78 not found: ID does not exist" containerID="a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"
Sep 17 19:42:51.497959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.497912    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78"} err="failed to get container status \"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78\": rpc error: code = NotFound desc = could not find container \"a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78\": container with ID starting with a460caf12ea3a430b09bf639acf299114ea3117638446e7f2353069171d1cc78 not found: ID does not exist"
Sep 17 19:42:51.497959 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.497927    2586 scope.go:117] "RemoveContainer" containerID="647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"
Sep 17 19:42:51.498123 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.498097    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0\": container with ID starting with 647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0 not found: ID does not exist" containerID="647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"
Sep 17 19:42:51.498167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.498125    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0"} err="failed to get container status \"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0\": rpc error: code = NotFound desc = could not find container \"647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0\": container with ID starting with 647bbf0fc26ff87f947fa3e82dbd1e708879435b4fa4076be992d2c9543fc6c0 not found: ID does not exist"
Sep 17 19:42:51.498167 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.498140    2586 scope.go:117] "RemoveContainer" containerID="4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"
Sep 17 19:42:51.498359 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:42:51.498342    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4\": container with ID starting with 4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4 not found: ID does not exist" containerID="4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"
Sep 17 19:42:51.498398 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:51.498366    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4"} err="failed to get container status \"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4\": rpc error: code = NotFound desc = could not find container \"4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4\": container with ID starting with 4b65d9ee3936177c45ed78a424a185fa559da5788be8849cce045497fea0f0f4 not found: ID does not exist"
Sep 17 19:42:53.016133 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:53.016102    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" path="/var/lib/kubelet/pods/39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe/volumes"
Sep 17 19:42:53.016582 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:53.016568    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" path="/var/lib/kubelet/pods/708b0e1c-35be-4816-ae39-f67640626b9b/volumes"
Sep 17 19:42:56.518928 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:56.518899    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:42:56.614915 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:42:56.614881    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:01.888083 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:43:01.888052    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:07.024071 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:43:07.023991    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:07.582533 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:43:07.582474    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:43:57.128576 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:43:57.128545    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:02.265987 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:02.265940    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:02.581425 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:02.581393    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:47.267363 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:47.267287    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:52.388188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:52.388149    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:52.467541 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:52.467507    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:52.522293 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:52.522264    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:57.539266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.539231    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:57.574931 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.574902    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:44:57.575197 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575186    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kube-rbac-proxy"
Sep 17 19:44:57.575240 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575199    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kube-rbac-proxy"
Sep 17 19:44:57.575240 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575208    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kube-rbac-proxy"
Sep 17 19:44:57.575240 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575215    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kube-rbac-proxy"
Sep 17 19:44:57.575240 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575221    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container"
Sep 17 19:44:57.575240 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575227    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575249    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="storage-initializer"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575256    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="storage-initializer"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575263    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="storage-initializer"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575268    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="storage-initializer"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575287    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575292    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575336    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kserve-container"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575342    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="39e280e1-aa16-49eb-b4f7-9c8f9af6a7fe" containerName="kube-rbac-proxy"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575348    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kserve-container"
Sep 17 19:44:57.575396 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.575354    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="708b0e1c-35be-4816-ae39-f67640626b9b" containerName="kube-rbac-proxy"
Sep 17 19:44:57.578312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.578296    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.580936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580847    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:44:57.580936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580847    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Sep 17 19:44:57.580936 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580929    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:44:57.581162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580896    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Sep 17 19:44:57.581162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580960    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:44:57.581162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.580937    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:44:57.581880 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.581861    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:44:57.587604 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.587584    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:44:57.667752 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.667724    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.667914 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.667787    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.667914 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.667843    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.667914 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.667876    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblzm\" (UniqueName: \"kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.668064 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.667913    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.671017 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.670985    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:44:57.769237 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769206    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblzm\" (UniqueName: \"kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.769378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769256    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.769378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769296    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.769378 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769361    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.769536 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769390    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.769813 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.769793    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.770213 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.770193    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.770257 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.770193    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.771880 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.771862    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.777390 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.777363    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblzm\" (UniqueName: \"kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-xm8vk\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:57.889183 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:57.889102    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:44:58.159162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:58.159138    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:44:58.161862 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:44:58.161832    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a92ffe_4ccf_41fe_a3dd_633748d0217e.slice/crio-1e3cb47b075de6e552a0e68b91dac0c2538b25ffcf5db5899807d2ba1c6d152d WatchSource:0}: Error finding container 1e3cb47b075de6e552a0e68b91dac0c2538b25ffcf5db5899807d2ba1c6d152d: Status 404 returned error can't find the container with id 1e3cb47b075de6e552a0e68b91dac0c2538b25ffcf5db5899807d2ba1c6d152d
Sep 17 19:44:58.978991 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:58.978949    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerStarted","Data":"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"}
Sep 17 19:44:58.979438 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:44:58.979052    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerStarted","Data":"1e3cb47b075de6e552a0e68b91dac0c2538b25ffcf5db5899807d2ba1c6d152d"}
Sep 17 19:45:02.997404 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:02.997375    2586 generic.go:356] "Generic (PLEG): container finished" podID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerID="a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec" exitCode=0
Sep 17 19:45:02.997783 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:02.997417    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerDied","Data":"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"}
Sep 17 19:45:02.998762 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:02.998747    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:45:22.099421 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:22.099395    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerStarted","Data":"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"}
Sep 17 19:45:23.105779 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:23.105737    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerStarted","Data":"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"}
Sep 17 19:45:23.106214 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:23.105804    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:45:23.106214 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:23.105836    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:45:23.107481 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:23.107450    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:45:23.126047 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:23.126002    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podStartSLOduration=2.987607995 podStartE2EDuration="26.125988699s" totalImagesPullingTime="19.020611247s" totalInitContainerRuntime="4.117769457s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:44:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:45:23.124530256 +0000 UTC m=+2368.290271563" watchObservedRunningTime="2026-09-17 19:45:23.125988699 +0000 UTC m=+2368.291729974"
Sep 17 19:45:24.111342 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:24.111309    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:45:29.115810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:29.115781    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:45:29.116266 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:29.116241    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:45:39.116218 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:39.116179    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:45:49.116583 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:49.116537    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:45:57.116738 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:57.116711    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:45:57.117996 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:57.117969    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:45:59.116227 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:45:59.116192    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:46:09.117141 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:09.117053    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:46:19.117195 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:19.117158    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:46:29.117085 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:29.117058    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:46:38.140367 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.140332    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:46:38.140860 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.140661    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" containerID="cri-o://fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826" gracePeriod=30
Sep 17 19:46:38.140860 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.140682    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kube-rbac-proxy" containerID="cri-o://bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9" gracePeriod=30
Sep 17 19:46:38.172972 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.172944    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:38.421112 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.421025    2586 generic.go:356] "Generic (PLEG): container finished" podID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerID="bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9" exitCode=2
Sep 17 19:46:38.421112 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:38.421073    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerDied","Data":"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"}
Sep 17 19:46:39.111590 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:39.111548    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Sep 17 19:46:39.116985 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:39.116948    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Sep 17 19:46:42.082212 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.082186    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:46:42.100750 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.100725    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config\") pod \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") "
Sep 17 19:46:42.100893 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.100756    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls\") pod \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") "
Sep 17 19:46:42.100893 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.100799    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert\") pod \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") "
Sep 17 19:46:42.100893 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.100850    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mblzm\" (UniqueName: \"kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm\") pod \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") "
Sep 17 19:46:42.101115 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.100918    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location\") pod \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\" (UID: \"26a92ffe-4ccf-41fe-a3dd-633748d0217e\") "
Sep 17 19:46:42.101448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.101406    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "26a92ffe-4ccf-41fe-a3dd-633748d0217e" (UID: "26a92ffe-4ccf-41fe-a3dd-633748d0217e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:46:42.101569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.101448    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "26a92ffe-4ccf-41fe-a3dd-633748d0217e" (UID: "26a92ffe-4ccf-41fe-a3dd-633748d0217e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:46:42.101569 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.101508    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "26a92ffe-4ccf-41fe-a3dd-633748d0217e" (UID: "26a92ffe-4ccf-41fe-a3dd-633748d0217e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:46:42.103508 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.103483    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm" (OuterVolumeSpecName: "kube-api-access-mblzm") pod "26a92ffe-4ccf-41fe-a3dd-633748d0217e" (UID: "26a92ffe-4ccf-41fe-a3dd-633748d0217e"). InnerVolumeSpecName "kube-api-access-mblzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:46:42.103679 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.103662    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "26a92ffe-4ccf-41fe-a3dd-633748d0217e" (UID: "26a92ffe-4ccf-41fe-a3dd-633748d0217e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:46:42.202540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.202485    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:46:42.202540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.202508    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/26a92ffe-4ccf-41fe-a3dd-633748d0217e-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:46:42.202540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.202519    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/26a92ffe-4ccf-41fe-a3dd-633748d0217e-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:46:42.202540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.202528    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mblzm\" (UniqueName: \"kubernetes.io/projected/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kube-api-access-mblzm\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:46:42.202540 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.202536    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/26a92ffe-4ccf-41fe-a3dd-633748d0217e-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:46:42.440051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.440018    2586 generic.go:356] "Generic (PLEG): container finished" podID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerID="fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826" exitCode=0
Sep 17 19:46:42.440202 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.440069    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"
Sep 17 19:46:42.440202 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.440099    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerDied","Data":"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"}
Sep 17 19:46:42.440202 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.440177    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk" event={"ID":"26a92ffe-4ccf-41fe-a3dd-633748d0217e","Type":"ContainerDied","Data":"1e3cb47b075de6e552a0e68b91dac0c2538b25ffcf5db5899807d2ba1c6d152d"}
Sep 17 19:46:42.440314 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.440203    2586 scope.go:117] "RemoveContainer" containerID="bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"
Sep 17 19:46:42.452233 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.452215    2586 scope.go:117] "RemoveContainer" containerID="fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"
Sep 17 19:46:42.460008 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.459991    2586 scope.go:117] "RemoveContainer" containerID="a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"
Sep 17 19:46:42.464660 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.464613    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:46:42.467331 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.467317    2586 scope.go:117] "RemoveContainer" containerID="bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"
Sep 17 19:46:42.467605 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:46:42.467585    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9\": container with ID starting with bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9 not found: ID does not exist" containerID="bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"
Sep 17 19:46:42.467701 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.467629    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9"} err="failed to get container status \"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9\": rpc error: code = NotFound desc = could not find container \"bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9\": container with ID starting with bec0dc2e44179c7ed839fcc5f230f17606180b662644b59ae8f4ecc79f1f24b9 not found: ID does not exist"
Sep 17 19:46:42.467701 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.467648    2586 scope.go:117] "RemoveContainer" containerID="fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"
Sep 17 19:46:42.467885 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:46:42.467868    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826\": container with ID starting with fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826 not found: ID does not exist" containerID="fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"
Sep 17 19:46:42.467923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.467890    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826"} err="failed to get container status \"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826\": rpc error: code = NotFound desc = could not find container \"fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826\": container with ID starting with fb28e9212f2f9b772d6103dcbec33a4492b7cc4b3d9de92a2369a1dadf058826 not found: ID does not exist"
Sep 17 19:46:42.467923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.467904    2586 scope.go:117] "RemoveContainer" containerID="a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"
Sep 17 19:46:42.468156 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:46:42.468141    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec\": container with ID starting with a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec not found: ID does not exist" containerID="a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"
Sep 17 19:46:42.468226 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.468159    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec"} err="failed to get container status \"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec\": rpc error: code = NotFound desc = could not find container \"a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec\": container with ID starting with a00f6fe479b7cc13d03c286c6a3729139eb9255bc3568714a5943cdd29d3cdec not found: ID does not exist"
Sep 17 19:46:42.470483 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:42.470453    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-xm8vk"]
Sep 17 19:46:43.016180 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:43.016148    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" path="/var/lib/kubelet/pods/26a92ffe-4ccf-41fe-a3dd-633748d0217e/volumes"
Sep 17 19:46:43.243473 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:43.243440    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:43.432198 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:43.432165    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:44.170118 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:44.170089    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:49.292187 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:49.292134    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:49.635598 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:49.635510    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:46:58.532586 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:46:58.532549    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:39.403291 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:39.403208    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:43.376180 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:43.376141    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:44.529820 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:44.529787    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:44.686038 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:44.685994    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:48.495458 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.495425    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:48.522826 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.522795    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:47:48.523111 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523099    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="storage-initializer"
Sep 17 19:47:48.523155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523113    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="storage-initializer"
Sep 17 19:47:48.523155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523131    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kube-rbac-proxy"
Sep 17 19:47:48.523155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523137    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kube-rbac-proxy"
Sep 17 19:47:48.523155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523145    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container"
Sep 17 19:47:48.523155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523151    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container"
Sep 17 19:47:48.523309 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523220    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kserve-container"
Sep 17 19:47:48.523309 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.523227    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a92ffe-4ccf-41fe-a3dd-633748d0217e" containerName="kube-rbac-proxy"
Sep 17 19:47:48.526139 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.526125    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.528607 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528586    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Sep 17 19:47:48.528764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528678    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:47:48.528764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528701    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:47:48.528764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528596    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:47:48.528764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528684    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Sep 17 19:47:48.528991 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.528801    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:47:48.529073 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.529057    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:47:48.536176 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.536156    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:47:48.603963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.603932    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.604099 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.603975    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.604099 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.604002    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm2sp\" (UniqueName: \"kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.604099 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.604081    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.604200 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.604144    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705282 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705255    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705300    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705337    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705353    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm2sp\" (UniqueName: \"kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705387 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705374    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.705831 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.705803    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.706037 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.706023    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.706082 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.706040    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.708411 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.708383    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.714790 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.714332    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm2sp\" (UniqueName: \"kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp\") pod \"isvc-xgboost-v2-predictor-fcc668bc-g9gl4\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:48.838345 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:48.838322    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:49.112350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:49.112327    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:47:49.114830 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:47:49.114799    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68284cca_83d4_400f_9b61_309479b24de3.slice/crio-b507fa80dc0f1a0c6cdc6bfacbc723dcbcf8a8bdb7f9a8c323972bc944b7cada WatchSource:0}: Error finding container b507fa80dc0f1a0c6cdc6bfacbc723dcbcf8a8bdb7f9a8c323972bc944b7cada: Status 404 returned error can't find the container with id b507fa80dc0f1a0c6cdc6bfacbc723dcbcf8a8bdb7f9a8c323972bc944b7cada
Sep 17 19:47:49.725054 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:49.725021    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerStarted","Data":"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"}
Sep 17 19:47:49.725425 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:49.725081    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerStarted","Data":"b507fa80dc0f1a0c6cdc6bfacbc723dcbcf8a8bdb7f9a8c323972bc944b7cada"}
Sep 17 19:47:49.783472 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:49.783437    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:47:53.743054 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:53.743023    2586 generic.go:356] "Generic (PLEG): container finished" podID="68284cca-83d4-400f-9b61-309479b24de3" containerID="d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1" exitCode=0
Sep 17 19:47:53.743484 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:53.743095    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerDied","Data":"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"}
Sep 17 19:47:54.749429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:54.749397    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerStarted","Data":"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"}
Sep 17 19:47:54.749931 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:54.749465    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerStarted","Data":"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"}
Sep 17 19:47:54.749931 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:54.749757    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:54.769456 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:54.769407    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podStartSLOduration=2.909950468 podStartE2EDuration="6.769393773s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.859443305s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:47:54.767557639 +0000 UTC m=+2519.933298957" watchObservedRunningTime="2026-09-17 19:47:54.769393773 +0000 UTC m=+2519.935135048"
Sep 17 19:47:55.754132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:55.754097    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:47:55.755605 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:55.755576    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:47:56.758110 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:47:56.758075    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:01.768567 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:01.768539    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:48:01.769022 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:01.768998    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:11.769729 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:11.769688    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:14.693465 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:14.693423    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:19.833302 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:19.833264    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:19.981810 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:19.981760    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:48:21.769105 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:21.769073    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:31.769639 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:31.769594    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:41.769497 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:41.769456    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:48:51.769682 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:48:51.769613    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:49:01.770068 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:01.770036    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:49:09.123210 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:09.123133    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:49:09.123589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:09.123400    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kube-rbac-proxy" containerID="cri-o://e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c" gracePeriod=30
Sep 17 19:49:09.123589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:09.123387    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" containerID="cri-o://3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54" gracePeriod=30
Sep 17 19:49:09.131837 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:09.131812    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:49:10.028554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:10.028519    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:49:10.073180 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:10.073148    2586 generic.go:356] "Generic (PLEG): container finished" podID="68284cca-83d4-400f-9b61-309479b24de3" containerID="e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c" exitCode=2
Sep 17 19:49:10.073336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:10.073220    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerDied","Data":"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"}
Sep 17 19:49:11.758578 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:11.758538    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Sep 17 19:49:11.769317 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:11.769292    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Sep 17 19:49:12.861169 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.861147    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:49:12.939887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.939859    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls\") pod \"68284cca-83d4-400f-9b61-309479b24de3\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") "
Sep 17 19:49:12.940051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.939945    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location\") pod \"68284cca-83d4-400f-9b61-309479b24de3\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") "
Sep 17 19:49:12.940051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.939980    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert\") pod \"68284cca-83d4-400f-9b61-309479b24de3\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") "
Sep 17 19:49:12.940051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.940012    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config\") pod \"68284cca-83d4-400f-9b61-309479b24de3\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") "
Sep 17 19:49:12.940051 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.940040    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm2sp\" (UniqueName: \"kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp\") pod \"68284cca-83d4-400f-9b61-309479b24de3\" (UID: \"68284cca-83d4-400f-9b61-309479b24de3\") "
Sep 17 19:49:12.940328 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.940301    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "68284cca-83d4-400f-9b61-309479b24de3" (UID: "68284cca-83d4-400f-9b61-309479b24de3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:12.940377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.940305    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "68284cca-83d4-400f-9b61-309479b24de3" (UID: "68284cca-83d4-400f-9b61-309479b24de3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:49:12.940377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.940349    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "68284cca-83d4-400f-9b61-309479b24de3" (UID: "68284cca-83d4-400f-9b61-309479b24de3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:49:12.942154 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.942135    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "68284cca-83d4-400f-9b61-309479b24de3" (UID: "68284cca-83d4-400f-9b61-309479b24de3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:49:12.942228 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:12.942210    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp" (OuterVolumeSpecName: "kube-api-access-vm2sp") pod "68284cca-83d4-400f-9b61-309479b24de3" (UID: "68284cca-83d4-400f-9b61-309479b24de3"). InnerVolumeSpecName "kube-api-access-vm2sp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:49:13.041506 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.041483    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/68284cca-83d4-400f-9b61-309479b24de3-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:49:13.041506 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.041506    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:49:13.041668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.041516    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/68284cca-83d4-400f-9b61-309479b24de3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:49:13.041668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.041524    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm2sp\" (UniqueName: \"kubernetes.io/projected/68284cca-83d4-400f-9b61-309479b24de3-kube-api-access-vm2sp\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:49:13.041668 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.041533    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/68284cca-83d4-400f-9b61-309479b24de3-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:49:13.087231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.087201    2586 generic.go:356] "Generic (PLEG): container finished" podID="68284cca-83d4-400f-9b61-309479b24de3" containerID="3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54" exitCode=0
Sep 17 19:49:13.087357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.087236    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerDied","Data":"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"}
Sep 17 19:49:13.087357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.087257    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"
Sep 17 19:49:13.087357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.087309    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4" event={"ID":"68284cca-83d4-400f-9b61-309479b24de3","Type":"ContainerDied","Data":"b507fa80dc0f1a0c6cdc6bfacbc723dcbcf8a8bdb7f9a8c323972bc944b7cada"}
Sep 17 19:49:13.087357 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.087349    2586 scope.go:117] "RemoveContainer" containerID="e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"
Sep 17 19:49:13.101035 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.101014    2586 scope.go:117] "RemoveContainer" containerID="3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"
Sep 17 19:49:13.106963 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.106941    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:49:13.109133 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.109117    2586 scope.go:117] "RemoveContainer" containerID="d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"
Sep 17 19:49:13.110607 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.110585    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-g9gl4"]
Sep 17 19:49:13.116712 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.116695    2586 scope.go:117] "RemoveContainer" containerID="e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"
Sep 17 19:49:13.116969 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:49:13.116950    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c\": container with ID starting with e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c not found: ID does not exist" containerID="e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"
Sep 17 19:49:13.117027 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.116982    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c"} err="failed to get container status \"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c\": rpc error: code = NotFound desc = could not find container \"e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c\": container with ID starting with e95f1e5145cf01b739de841393e01f60dedff193ea3afed3fc745310959e585c not found: ID does not exist"
Sep 17 19:49:13.117027 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.116999    2586 scope.go:117] "RemoveContainer" containerID="3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"
Sep 17 19:49:13.117231 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:49:13.117215    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54\": container with ID starting with 3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54 not found: ID does not exist" containerID="3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"
Sep 17 19:49:13.117285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.117235    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54"} err="failed to get container status \"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54\": rpc error: code = NotFound desc = could not find container \"3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54\": container with ID starting with 3a7565fa51ad6f57ea7a0139d840163492cc7e6a93c354d7c289dfcc410c5a54 not found: ID does not exist"
Sep 17 19:49:13.117285 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.117249    2586 scope.go:117] "RemoveContainer" containerID="d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"
Sep 17 19:49:13.117521 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:49:13.117504    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1\": container with ID starting with d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1 not found: ID does not exist" containerID="d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"
Sep 17 19:49:13.117574 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:13.117524    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1"} err="failed to get container status \"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1\": rpc error: code = NotFound desc = could not find container \"d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1\": container with ID starting with d652aeeced21839a7b762f66bf6e7bc0ab8b6dcbf25360745fa8ba05493a2fa1 not found: ID does not exist"
Sep 17 19:49:14.283653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:14.283594    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:49:14.471185 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:14.471150    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:49:15.016562 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:15.016526    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68284cca-83d4-400f-9b61-309479b24de3" path="/var/lib/kubelet/pods/68284cca-83d4-400f-9b61-309479b24de3/volumes"
Sep 17 19:49:15.101712 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:15.101682    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:49:15.259155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:49:15.259124    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:35.185245 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:35.185205    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:40.329280 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:40.329206    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:40.915998 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:40.915965    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:44.878123 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:44.878095    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:45.030874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.030844    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:45.068392 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068360    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:50:45.068682 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068668    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container"
Sep 17 19:50:45.068739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068684    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container"
Sep 17 19:50:45.068739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068696    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kube-rbac-proxy"
Sep 17 19:50:45.068739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068702    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kube-rbac-proxy"
Sep 17 19:50:45.068739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068717    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="storage-initializer"
Sep 17 19:50:45.068739 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068722    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="storage-initializer"
Sep 17 19:50:45.068897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068770    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kserve-container"
Sep 17 19:50:45.068897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.068777    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="68284cca-83d4-400f-9b61-309479b24de3" containerName="kube-rbac-proxy"
Sep 17 19:50:45.071907 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.071885    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.074202 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.074183    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:50:45.074373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.074217    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:50:45.074373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.074310    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:50:45.074479 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.074384    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:50:45.075080 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.075058    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-predictor-serving-cert\""
Sep 17 19:50:45.075184 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.075094    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:50:45.075184 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.075124    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-global-pass-kube-rbac-proxy-sar-config\""
Sep 17 19:50:45.084064 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.084045    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:50:45.114676 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.114646    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.114827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.114712    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.114827 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.114749    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.114944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.114836    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.114944 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.114900    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.138897 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.138828    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:50:45.216028 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216001    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216151 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216045    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216218 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216168    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216274 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216219    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216274 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216266    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216610 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216587    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216716    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.216899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.216885    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.218892 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.218873    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.224681 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.224660    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb\") pod \"isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.382521 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.382494    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:45.655981 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:45.655899    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:50:45.659689 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:50:45.659655    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad74731_c1e1_476f_9012_5b97a30a2508.slice/crio-ea504de95cedaf933532a3cbd5ba541c90173bdbb71b939e6602ea37598d5645 WatchSource:0}: Error finding container ea504de95cedaf933532a3cbd5ba541c90173bdbb71b939e6602ea37598d5645: Status 404 returned error can't find the container with id ea504de95cedaf933532a3cbd5ba541c90173bdbb71b939e6602ea37598d5645
Sep 17 19:50:46.486946 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:46.486909    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerStarted","Data":"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"}
Sep 17 19:50:46.487417 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:46.486976    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerStarted","Data":"ea504de95cedaf933532a3cbd5ba541c90173bdbb71b939e6602ea37598d5645"}
Sep 17 19:50:47.492522 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:47.492486    2586 generic.go:356] "Generic (PLEG): container finished" podID="bad74731-c1e1-476f-9012-5b97a30a2508" containerID="8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b" exitCode=0
Sep 17 19:50:47.492969 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:47.492552    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerDied","Data":"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"}
Sep 17 19:50:48.498875 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.498835    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerStarted","Data":"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"}
Sep 17 19:50:48.499370 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.498906    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerStarted","Data":"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"}
Sep 17 19:50:48.499370 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.498940    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:48.499370 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.498969    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:48.500449 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.500418    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:50:48.520693 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:48.520652    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podStartSLOduration=2.45009037 podStartE2EDuration="3.520636779s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.070546409s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:50:48.518368181 +0000 UTC m=+2693.684109477" watchObservedRunningTime="2026-09-17 19:50:48.520636779 +0000 UTC m=+2693.686378053"
Sep 17 19:50:49.503532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:49.503496    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:50:54.507501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:54.507469    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:50:54.508058 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:54.508032    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:50:57.141642 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:57.141598    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:50:57.143376 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:50:57.143358    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:51:04.508751 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:04.508709    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:14.508769 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:14.508732    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:24.508329 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:24.508289    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:34.508232 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:34.508193    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:44.508575 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:44.508529    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:51.798376 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:51.798340    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:51:54.508510 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:54.508469    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Sep 17 19:51:56.918363 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:56.918325    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:51:57.076435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:51:57.076397    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:04.508439 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:04.508406    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:52:05.071730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.071697    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:52:05.072039 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.071988    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container" containerID="cri-o://f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729" gracePeriod=30
Sep 17 19:52:05.072103 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.072013    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kube-rbac-proxy" containerID="cri-o://de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5" gracePeriod=30
Sep 17 19:52:05.080400 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.080379    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:05.842108 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.842028    2586 generic.go:356] "Generic (PLEG): container finished" podID="bad74731-c1e1-476f-9012-5b97a30a2508" containerID="de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5" exitCode=2
Sep 17 19:52:05.842108 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:05.842085    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerDied","Data":"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"}
Sep 17 19:52:06.138905 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:06.138823    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:06.859250 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:06.859213    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:09.504451 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:09.504413    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Sep 17 19:52:10.018209 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.018180    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:52:10.083939 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.083855    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls\") pod \"bad74731-c1e1-476f-9012-5b97a30a2508\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") "
Sep 17 19:52:10.083939 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.083911    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert\") pod \"bad74731-c1e1-476f-9012-5b97a30a2508\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") "
Sep 17 19:52:10.083939 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.083939    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb\") pod \"bad74731-c1e1-476f-9012-5b97a30a2508\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") "
Sep 17 19:52:10.084186 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.083974    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location\") pod \"bad74731-c1e1-476f-9012-5b97a30a2508\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") "
Sep 17 19:52:10.084186 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.084017    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config\") pod \"bad74731-c1e1-476f-9012-5b97a30a2508\" (UID: \"bad74731-c1e1-476f-9012-5b97a30a2508\") "
Sep 17 19:52:10.084312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.084281    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bad74731-c1e1-476f-9012-5b97a30a2508" (UID: "bad74731-c1e1-476f-9012-5b97a30a2508"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:52:10.084312 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.084292    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bad74731-c1e1-476f-9012-5b97a30a2508" (UID: "bad74731-c1e1-476f-9012-5b97a30a2508"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:52:10.084449 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.084401    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bad74731-c1e1-476f-9012-5b97a30a2508" (UID: "bad74731-c1e1-476f-9012-5b97a30a2508"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:52:10.086076 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.086054    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bad74731-c1e1-476f-9012-5b97a30a2508" (UID: "bad74731-c1e1-476f-9012-5b97a30a2508"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:52:10.086225 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.086205    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb" (OuterVolumeSpecName: "kube-api-access-mhlgb") pod "bad74731-c1e1-476f-9012-5b97a30a2508" (UID: "bad74731-c1e1-476f-9012-5b97a30a2508"). InnerVolumeSpecName "kube-api-access-mhlgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:52:10.185554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.185519    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bad74731-c1e1-476f-9012-5b97a30a2508-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:52:10.185554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.185547    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:52:10.185554 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.185556    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhlgb\" (UniqueName: \"kubernetes.io/projected/bad74731-c1e1-476f-9012-5b97a30a2508-kube-api-access-mhlgb\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:52:10.185796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.185565    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bad74731-c1e1-476f-9012-5b97a30a2508-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:52:10.185796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.185574    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bad74731-c1e1-476f-9012-5b97a30a2508-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:52:10.864949 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.864910    2586 generic.go:356] "Generic (PLEG): container finished" podID="bad74731-c1e1-476f-9012-5b97a30a2508" containerID="f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729" exitCode=0
Sep 17 19:52:10.865406 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.864954    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerDied","Data":"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"}
Sep 17 19:52:10.865406 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.864966    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"
Sep 17 19:52:10.865406 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.865014    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf" event={"ID":"bad74731-c1e1-476f-9012-5b97a30a2508","Type":"ContainerDied","Data":"ea504de95cedaf933532a3cbd5ba541c90173bdbb71b939e6602ea37598d5645"}
Sep 17 19:52:10.865406 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.865052    2586 scope.go:117] "RemoveContainer" containerID="de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"
Sep 17 19:52:10.873653 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.873610    2586 scope.go:117] "RemoveContainer" containerID="f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"
Sep 17 19:52:10.883797 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.883661    2586 scope.go:117] "RemoveContainer" containerID="8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"
Sep 17 19:52:10.889871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.889829    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:52:10.891885 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.891745    2586 scope.go:117] "RemoveContainer" containerID="de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"
Sep 17 19:52:10.892142 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:52:10.892111    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5\": container with ID starting with de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5 not found: ID does not exist" containerID="de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"
Sep 17 19:52:10.892241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.892156    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5"} err="failed to get container status \"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5\": rpc error: code = NotFound desc = could not find container \"de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5\": container with ID starting with de91d6e2a36e9a8fb2b35a6c0f17ccdf56dada545a9ee1e731dab5a7d1e0cdb5 not found: ID does not exist"
Sep 17 19:52:10.892241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.892179    2586 scope.go:117] "RemoveContainer" containerID="f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"
Sep 17 19:52:10.892501 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:52:10.892470    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729\": container with ID starting with f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729 not found: ID does not exist" containerID="f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"
Sep 17 19:52:10.892612 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.892508    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729"} err="failed to get container status \"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729\": rpc error: code = NotFound desc = could not find container \"f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729\": container with ID starting with f3f1f924df04a7435f3d7d1fe1c22ba2f4ebb87fa0aae6254962048ae2d59729 not found: ID does not exist"
Sep 17 19:52:10.892612 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.892532    2586 scope.go:117] "RemoveContainer" containerID="8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"
Sep 17 19:52:10.892936 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:52:10.892906    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b\": container with ID starting with 8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b not found: ID does not exist" containerID="8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"
Sep 17 19:52:10.893022 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.892943    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b"} err="failed to get container status \"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b\": rpc error: code = NotFound desc = could not find container \"8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b\": container with ID starting with 8bb733ccc7bb72b7a3afebbe8f0e1a22b632acc18d0b1f87474988f09206a93b not found: ID does not exist"
Sep 17 19:52:10.894150 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:10.894134    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-global-pass-predictor-b6b766889-p45kf"]
Sep 17 19:52:11.014978 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:11.014944    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" path="/var/lib/kubelet/pods/bad74731-c1e1-476f-9012-5b97a30a2508/volumes"
Sep 17 19:52:16.169223 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:16.169188    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:17.275390 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.275353    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:17.314589 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314560    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:52:17.314878 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314866    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kube-rbac-proxy"
Sep 17 19:52:17.314923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314879    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kube-rbac-proxy"
Sep 17 19:52:17.314923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314894    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container"
Sep 17 19:52:17.314923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314899    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container"
Sep 17 19:52:17.314923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314911    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="storage-initializer"
Sep 17 19:52:17.314923 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314916    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="storage-initializer"
Sep 17 19:52:17.315078 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314963    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kube-rbac-proxy"
Sep 17 19:52:17.315078 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.314969    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="bad74731-c1e1-476f-9012-5b97a30a2508" containerName="kserve-container"
Sep 17 19:52:17.319822 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.319801    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.322209 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.322187    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:52:17.322325 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.322251    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:52:17.322471 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.322458    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:52:17.322735 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.322723    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:52:17.323265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.323247    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-kube-rbac-proxy-sar-config\""
Sep 17 19:52:17.323320 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.323255    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:52:17.323320 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.323291    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-pass-predictor-serving-cert\""
Sep 17 19:52:17.328971 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.328950    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:52:17.441706 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.441678    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.441874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.441741    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhjph\" (UniqueName: \"kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.441874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.441793    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.441874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.441820    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.441874 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.441845    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.442221 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.442196    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:17.542982 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.542949    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhjph\" (UniqueName: \"kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543148 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.542993    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543148 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543013    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543148 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543034    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543148 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543071    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543358    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543838 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543818    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.543887 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.543818    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.545572 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.545556    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.552360 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.552341    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhjph\" (UniqueName: \"kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph\") pod \"isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.631788 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.631757    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:17.904255 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:17.904126    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:52:17.906645 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:52:17.906605    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa272174_40eb_4392_b6c9_fb0591307ef8.slice/crio-b717d3e15530c11008e746218763f6a6c2767c7b1ed7d32a00e89e112c33808b WatchSource:0}: Error finding container b717d3e15530c11008e746218763f6a6c2767c7b1ed7d32a00e89e112c33808b: Status 404 returned error can't find the container with id b717d3e15530c11008e746218763f6a6c2767c7b1ed7d32a00e89e112c33808b
Sep 17 19:52:18.902980 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:18.902942    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerStarted","Data":"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"}
Sep 17 19:52:18.903372 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:18.903029    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerStarted","Data":"b717d3e15530c11008e746218763f6a6c2767c7b1ed7d32a00e89e112c33808b"}
Sep 17 19:52:19.908999 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:19.908954    2586 generic.go:356] "Generic (PLEG): container finished" podID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerID="566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517" exitCode=0
Sep 17 19:52:19.909437 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:19.909067    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerDied","Data":"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"}
Sep 17 19:52:20.915412 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.915375    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerStarted","Data":"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"}
Sep 17 19:52:20.915857 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.915438    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:20.915857 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.915470    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerStarted","Data":"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"}
Sep 17 19:52:20.915857 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.915615    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:20.917130 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.917105    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:20.938791 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:20.938746    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podStartSLOduration=2.8460060560000002 podStartE2EDuration="3.938730918s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.092724862s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:52:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:52:20.936531125 +0000 UTC m=+2786.102272431" watchObservedRunningTime="2026-09-17 19:52:20.938730918 +0000 UTC m=+2786.104472192"
Sep 17 19:52:21.920336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:21.920300    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:23.754098 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:23.754067    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:26.924924 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:26.924896    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:52:26.925417 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:26.925392    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:36.925513 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:36.925466    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:46.925861 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:46.925816    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:55.430448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:55.430415    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:52:56.926124 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:56.926081    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:52:58.385945 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:52:58.385907    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:03.519597 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:03.519563    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:04.424467 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:04.424428    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:06.925421 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:06.925383    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:53:16.925380 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:16.925341    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:53:26.925771 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:26.925723    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Sep 17 19:53:36.926466 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:36.926392    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:53:37.333495 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:37.333459    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:53:37.333698 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:37.333676    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container" containerID="cri-o://b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8" gracePeriod=30
Sep 17 19:53:37.333764 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:37.333727    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kube-rbac-proxy" containerID="cri-o://26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f" gracePeriod=30
Sep 17 19:53:37.336818 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:37.336787    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:38.239759 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:38.239726    2586 generic.go:356] "Generic (PLEG): container finished" podID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerID="26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f" exitCode=2
Sep 17 19:53:38.239759 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:38.239762    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerDied","Data":"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"}
Sep 17 19:53:38.370981 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:38.370953    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:38.586761 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:38.586728    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:41.878031 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:41.878010    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:53:42.000039 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:41.999956    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert\") pod \"fa272174-40eb-4392-b6c9-fb0591307ef8\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") "
Sep 17 19:53:42.000039 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000019    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dhjph\" (UniqueName: \"kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph\") pod \"fa272174-40eb-4392-b6c9-fb0591307ef8\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") "
Sep 17 19:53:42.000268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000063    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location\") pod \"fa272174-40eb-4392-b6c9-fb0591307ef8\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") "
Sep 17 19:53:42.000268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000088    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config\") pod \"fa272174-40eb-4392-b6c9-fb0591307ef8\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") "
Sep 17 19:53:42.000268 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000108    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls\") pod \"fa272174-40eb-4392-b6c9-fb0591307ef8\" (UID: \"fa272174-40eb-4392-b6c9-fb0591307ef8\") "
Sep 17 19:53:42.000435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000373    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa272174-40eb-4392-b6c9-fb0591307ef8" (UID: "fa272174-40eb-4392-b6c9-fb0591307ef8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:53:42.000435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000403    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa272174-40eb-4392-b6c9-fb0591307ef8" (UID: "fa272174-40eb-4392-b6c9-fb0591307ef8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:53:42.000512 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.000438    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa272174-40eb-4392-b6c9-fb0591307ef8" (UID: "fa272174-40eb-4392-b6c9-fb0591307ef8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:53:42.002262 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.002241    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph" (OuterVolumeSpecName: "kube-api-access-dhjph") pod "fa272174-40eb-4392-b6c9-fb0591307ef8" (UID: "fa272174-40eb-4392-b6c9-fb0591307ef8"). InnerVolumeSpecName "kube-api-access-dhjph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:53:42.002350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.002316    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa272174-40eb-4392-b6c9-fb0591307ef8" (UID: "fa272174-40eb-4392-b6c9-fb0591307ef8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:53:42.101055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.101030    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:53:42.101055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.101051    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa272174-40eb-4392-b6c9-fb0591307ef8-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:53:42.101055 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.101063    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa272174-40eb-4392-b6c9-fb0591307ef8-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:53:42.101251 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.101071    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhjph\" (UniqueName: \"kubernetes.io/projected/fa272174-40eb-4392-b6c9-fb0591307ef8-kube-api-access-dhjph\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:53:42.101251 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.101079    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa272174-40eb-4392-b6c9-fb0591307ef8-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:53:42.258606 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.258530    2586 generic.go:356] "Generic (PLEG): container finished" podID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerID="b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8" exitCode=0
Sep 17 19:53:42.258606 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.258588    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"
Sep 17 19:53:42.258796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.258607    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerDied","Data":"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"}
Sep 17 19:53:42.258796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.258685    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd" event={"ID":"fa272174-40eb-4392-b6c9-fb0591307ef8","Type":"ContainerDied","Data":"b717d3e15530c11008e746218763f6a6c2767c7b1ed7d32a00e89e112c33808b"}
Sep 17 19:53:42.258796 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.258714    2586 scope.go:117] "RemoveContainer" containerID="26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"
Sep 17 19:53:42.267216 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.267201    2586 scope.go:117] "RemoveContainer" containerID="b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"
Sep 17 19:53:42.274457 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.274440    2586 scope.go:117] "RemoveContainer" containerID="566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"
Sep 17 19:53:42.280945 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.280924    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:53:42.281612 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.281600    2586 scope.go:117] "RemoveContainer" containerID="26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"
Sep 17 19:53:42.281890 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:53:42.281870    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f\": container with ID starting with 26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f not found: ID does not exist" containerID="26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"
Sep 17 19:53:42.281957 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.281898    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f"} err="failed to get container status \"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f\": rpc error: code = NotFound desc = could not find container \"26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f\": container with ID starting with 26782e2ad611079f48bfe95dcff4d3e6bc34d52788620016da941f5b9328277f not found: ID does not exist"
Sep 17 19:53:42.281957 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.281915    2586 scope.go:117] "RemoveContainer" containerID="b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"
Sep 17 19:53:42.282137 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:53:42.282121    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8\": container with ID starting with b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8 not found: ID does not exist" containerID="b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"
Sep 17 19:53:42.282182 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.282143    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8"} err="failed to get container status \"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8\": rpc error: code = NotFound desc = could not find container \"b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8\": container with ID starting with b2e3c7a86b2238d85799065541bf9b6dce9c04149ec16bcf503ff2bbdf7827e8 not found: ID does not exist"
Sep 17 19:53:42.282182 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.282158    2586 scope.go:117] "RemoveContainer" containerID="566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"
Sep 17 19:53:42.282398 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:53:42.282384    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517\": container with ID starting with 566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517 not found: ID does not exist" containerID="566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"
Sep 17 19:53:42.282448 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.282399    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517"} err="failed to get container status \"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517\": rpc error: code = NotFound desc = could not find container \"566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517\": container with ID starting with 566bf02583fc6dc995d2563e6dd5e62e7a7f4deb3957f4a80ccd55c1fca4f517 not found: ID does not exist"
Sep 17 19:53:42.289527 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:42.287161    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-pass-predictor-784495866f-lhwvd"]
Sep 17 19:53:43.016081 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:43.016048    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" path="/var/lib/kubelet/pods/fa272174-40eb-4392-b6c9-fb0591307ef8/volumes"
Sep 17 19:53:48.402079 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:48.402043    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:49.503645 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.503563    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:49.544104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544068    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:53:49.544383 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544371    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kube-rbac-proxy"
Sep 17 19:53:49.544427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544384    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kube-rbac-proxy"
Sep 17 19:53:49.544427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544393    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container"
Sep 17 19:53:49.544427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544399    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container"
Sep 17 19:53:49.544427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544407    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="storage-initializer"
Sep 17 19:53:49.544427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544412    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="storage-initializer"
Sep 17 19:53:49.544583 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544467    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kube-rbac-proxy"
Sep 17 19:53:49.544583 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.544474    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa272174-40eb-4392-b6c9-fb0591307ef8" containerName="kserve-container"
Sep 17 19:53:49.549125 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.549104    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.552133 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.552111    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-kube-rbac-proxy-sar-config\""
Sep 17 19:53:49.552243 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.552132    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Sep 17 19:53:49.552499 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.552479    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:53:49.553231 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.553213    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:53:49.553321 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.553241    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:53:49.553321 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.553284    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert\""
Sep 17 19:53:49.553429 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.553322    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:53:49.560087 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.560067    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:53:49.659317 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.659288    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.659501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.659336    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.659501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.659436    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.659501 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.659470    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.659680 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.659509    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6l2c\" (UniqueName: \"kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760723 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.760637    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760723 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.760679    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6l2c\" (UniqueName: \"kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760723 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.760721    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760981 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.760748    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760981 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.760779    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.760981 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:53:49.760900    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 19:53:49.760981 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:53:49.760979    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls podName:1b49878b-d586-4bbf-bb9b-468debf1e1b6 nodeName:}" failed. No retries permitted until 2026-09-17 19:53:50.260956907 +0000 UTC m=+2875.426698160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls") pod "isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6") : secret "isvc-sklearn-s3-tls-serving-pass-predictor-serving-cert" not found
Sep 17 19:53:49.761174 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.761039    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.761328 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.761312    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.761419 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.761402    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:49.769802 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:49.769782    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6l2c\" (UniqueName: \"kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:50.005212 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:50.005181    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:53:50.265672 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:50.265609    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:50.268267 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:50.268243    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:50.459561 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:50.459527    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:50.740700 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:50.740671    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:53:50.744034 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:53:50.744006    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b49878b_d586_4bbf_bb9b_468debf1e1b6.slice/crio-f483b576fe8a097486d10f85c77da56aaf8b789ef71092cdd11ea638927c8551 WatchSource:0}: Error finding container f483b576fe8a097486d10f85c77da56aaf8b789ef71092cdd11ea638927c8551: Status 404 returned error can't find the container with id f483b576fe8a097486d10f85c77da56aaf8b789ef71092cdd11ea638927c8551
Sep 17 19:53:51.302830 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:51.302791    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerStarted","Data":"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"}
Sep 17 19:53:51.302986 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:51.302875    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerStarted","Data":"f483b576fe8a097486d10f85c77da56aaf8b789ef71092cdd11ea638927c8551"}
Sep 17 19:53:52.308503 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:52.308463    2586 generic.go:356] "Generic (PLEG): container finished" podID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerID="ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379" exitCode=0
Sep 17 19:53:52.308899 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:52.308566    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerDied","Data":"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"}
Sep 17 19:53:53.314964 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.314928    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerStarted","Data":"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"}
Sep 17 19:53:53.315416 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.314997    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:53.315416 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.315027    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:53.315416 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.315055    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerStarted","Data":"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"}
Sep 17 19:53:53.316377 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.316352    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:53:53.337749 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:53.337704    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podStartSLOduration=3.273960261 podStartE2EDuration="4.337690913s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.063730652s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:53:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:53:53.336151478 +0000 UTC m=+2878.501892773" watchObservedRunningTime="2026-09-17 19:53:53.337690913 +0000 UTC m=+2878.503432190"
Sep 17 19:53:54.319132 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:54.319100    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:53:59.324239 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:59.324207    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:53:59.324755 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:53:59.324727    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:05.185753 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:05.185714    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:54:06.555454 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:06.555420    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:54:09.325149 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:09.325104    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:10.417886 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:10.417857    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:54:10.551863 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:10.551826    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:54:19.324902 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:19.324855    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:29.325686 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:29.325613    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:39.325715 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:39.325670    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:49.325147 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:49.325106    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:54:59.325059 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:54:59.325020    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Sep 17 19:55:09.326564 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:09.326487    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:55:19.544188 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.544153    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:55:19.544798 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.544464    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container" containerID="cri-o://a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c" gracePeriod=30
Sep 17 19:55:19.544798 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.544523    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kube-rbac-proxy" containerID="cri-o://5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122" gracePeriod=30
Sep 17 19:55:19.548684 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.548647    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:19.678995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.678969    2586 generic.go:356] "Generic (PLEG): container finished" podID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerID="5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122" exitCode=2
Sep 17 19:55:19.679129 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:19.679047    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerDied","Data":"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"}
Sep 17 19:55:20.629561 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:20.629524    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:20.888600 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:20.888507    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:24.090712 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.090690    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:55:24.196155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196085    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config\") pod \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") "
Sep 17 19:55:24.196155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196120    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location\") pod \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") "
Sep 17 19:55:24.196155 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196154    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert\") pod \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") "
Sep 17 19:55:24.196427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196202    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s6l2c\" (UniqueName: \"kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c\") pod \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") "
Sep 17 19:55:24.196427 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196340    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") pod \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\" (UID: \"1b49878b-d586-4bbf-bb9b-468debf1e1b6\") "
Sep 17 19:55:24.196536 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196429    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1b49878b-d586-4bbf-bb9b-468debf1e1b6" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:55:24.196536 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196487    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b49878b-d586-4bbf-bb9b-468debf1e1b6" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:24.196651 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196533    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1b49878b-d586-4bbf-bb9b-468debf1e1b6" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:55:24.196651 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196640    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:55:24.196726 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196660    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:55:24.196726 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.196674    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1b49878b-d586-4bbf-bb9b-468debf1e1b6-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:55:24.198350 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.198331    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c" (OuterVolumeSpecName: "kube-api-access-s6l2c") pod "1b49878b-d586-4bbf-bb9b-468debf1e1b6" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6"). InnerVolumeSpecName "kube-api-access-s6l2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:55:24.198463 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.198397    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b49878b-d586-4bbf-bb9b-468debf1e1b6" (UID: "1b49878b-d586-4bbf-bb9b-468debf1e1b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:55:24.297532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.297507    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b49878b-d586-4bbf-bb9b-468debf1e1b6-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:55:24.297532 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.297532    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6l2c\" (UniqueName: \"kubernetes.io/projected/1b49878b-d586-4bbf-bb9b-468debf1e1b6-kube-api-access-s6l2c\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:55:24.704821 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.704786    2586 generic.go:356] "Generic (PLEG): container finished" podID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerID="a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c" exitCode=0
Sep 17 19:55:24.704984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.704846    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"
Sep 17 19:55:24.704984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.704849    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerDied","Data":"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"}
Sep 17 19:55:24.704984 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.704967    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2" event={"ID":"1b49878b-d586-4bbf-bb9b-468debf1e1b6","Type":"ContainerDied","Data":"f483b576fe8a097486d10f85c77da56aaf8b789ef71092cdd11ea638927c8551"}
Sep 17 19:55:24.705110 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.704993    2586 scope.go:117] "RemoveContainer" containerID="5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"
Sep 17 19:55:24.713634 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.713608    2586 scope.go:117] "RemoveContainer" containerID="a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"
Sep 17 19:55:24.721435 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.721412    2586 scope.go:117] "RemoveContainer" containerID="ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"
Sep 17 19:55:24.726956 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.726934    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:55:24.731454 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.731435    2586 scope.go:117] "RemoveContainer" containerID="5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"
Sep 17 19:55:24.731837 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:55:24.731816    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122\": container with ID starting with 5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122 not found: ID does not exist" containerID="5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"
Sep 17 19:55:24.731911 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.731849    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122"} err="failed to get container status \"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122\": rpc error: code = NotFound desc = could not find container \"5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122\": container with ID starting with 5a16728499077612840dfa289b7bd04bbef1ab3689788f73cbe7f485142bd122 not found: ID does not exist"
Sep 17 19:55:24.731911 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.731868    2586 scope.go:117] "RemoveContainer" containerID="a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"
Sep 17 19:55:24.732170 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:55:24.732152    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c\": container with ID starting with a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c not found: ID does not exist" containerID="a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"
Sep 17 19:55:24.732241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.732182    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c"} err="failed to get container status \"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c\": rpc error: code = NotFound desc = could not find container \"a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c\": container with ID starting with a34256380a43d3c8cee7364415014ffee6a041947a776b7e626e373e14f80f9c not found: ID does not exist"
Sep 17 19:55:24.732241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.732211    2586 scope.go:117] "RemoveContainer" containerID="ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"
Sep 17 19:55:24.732479 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:55:24.732463    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379\": container with ID starting with ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379 not found: ID does not exist" containerID="ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"
Sep 17 19:55:24.732522 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.732485    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379"} err="failed to get container status \"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379\": rpc error: code = NotFound desc = could not find container \"ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379\": container with ID starting with ae9374d908a90e13911d66d0aeee3aa08f14b17b7fb94310364911681d23b379 not found: ID does not exist"
Sep 17 19:55:24.732652 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:24.732634    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-pass-predictor-7455cc4b9d-r7cp2"]
Sep 17 19:55:25.015723 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:25.015637    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" path="/var/lib/kubelet/pods/1b49878b-d586-4bbf-bb9b-468debf1e1b6/volumes"
Sep 17 19:55:30.633241 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:30.633195    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:31.765162 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.765109    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:31.808585 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808547    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:55:31.808945 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808927    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kube-rbac-proxy"
Sep 17 19:55:31.809043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808948    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kube-rbac-proxy"
Sep 17 19:55:31.809043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808971    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container"
Sep 17 19:55:31.809043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808980    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container"
Sep 17 19:55:31.809043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.808997    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="storage-initializer"
Sep 17 19:55:31.809043 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.809007    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="storage-initializer"
Sep 17 19:55:31.809304 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.809073    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kserve-container"
Sep 17 19:55:31.809304 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.809085    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b49878b-d586-4bbf-bb9b-468debf1e1b6" containerName="kube-rbac-proxy"
Sep 17 19:55:31.815324 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.814972    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.817560 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.817490    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Sep 17 19:55:31.817560 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.817553    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Sep 17 19:55:31.817786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.817557    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Sep 17 19:55:31.817786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.817559    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-5mbcr\""
Sep 17 19:55:31.817786 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.817783    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Sep 17 19:55:31.818568 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.818540    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Sep 17 19:55:31.818696 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.818599    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Sep 17 19:55:31.825492 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.825472    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:55:31.858603 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.858580    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.858730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.858613    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.858730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.858664    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.858730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.858689    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcf4p\" (UniqueName: \"kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.858730 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.858712    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959165 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959130    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959170    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959306 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959193    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959306 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:55:31.959280    2586 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-serving-cert: secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 19:55:31.959450 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959317    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcf4p\" (UniqueName: \"kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959450 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:55:31.959354    2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls podName:4814cb8d-e65c-42bd-8762-64fb07ae87e5 nodeName:}" failed. No retries permitted until 2026-09-17 19:55:32.459332406 +0000 UTC m=+2977.625073660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls") pod "isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5") : secret "isvc-pmml-v2-kserve-predictor-serving-cert" not found
Sep 17 19:55:31.959450 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959401    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959806 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959785    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959901 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959834    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.959997 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.959976    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:31.967946 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:31.967925    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcf4p\" (UniqueName: \"kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:32.154593 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:32.154565    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:55:32.464126 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:32.464037    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:32.466579 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:32.466551    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:32.727104 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:32.727023    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:32.997403 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:32.997381    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:55:32.999772 ip-10-0-134-45 kubenswrapper[2586]: W0917 19:55:32.999740    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4814cb8d_e65c_42bd_8762_64fb07ae87e5.slice/crio-11bc324d60e3b9da3841345e274cd005403888a58f5f2f97d6da7a1809e004ce WatchSource:0}: Error finding container 11bc324d60e3b9da3841345e274cd005403888a58f5f2f97d6da7a1809e004ce: Status 404 returned error can't find the container with id 11bc324d60e3b9da3841345e274cd005403888a58f5f2f97d6da7a1809e004ce
Sep 17 19:55:33.747762 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:33.747727    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerStarted","Data":"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"}
Sep 17 19:55:33.747940 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:33.747793    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerStarted","Data":"11bc324d60e3b9da3841345e274cd005403888a58f5f2f97d6da7a1809e004ce"}
Sep 17 19:55:37.765044 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:37.765012    2586 generic.go:356] "Generic (PLEG): container finished" podID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerID="8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe" exitCode=0
Sep 17 19:55:37.765463 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:37.765080    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerDied","Data":"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"}
Sep 17 19:55:37.766229 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:37.766211    2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 19:55:45.811844 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.811805    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerStarted","Data":"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"}
Sep 17 19:55:45.812325 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.811867    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerStarted","Data":"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"}
Sep 17 19:55:45.812325 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.812134    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:45.812325 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.812279    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:45.813573 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.813547    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:55:45.835704 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:45.835663    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podStartSLOduration=3.655987965 podStartE2EDuration="14.835651372s" totalImagesPullingTime="7.270547975s" totalInitContainerRuntime="3.909115432s" isStatefulPod=false podCreationTimestamp="2026-09-17 19:55:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 19:55:45.834728403 +0000 UTC m=+2991.000469706" watchObservedRunningTime="2026-09-17 19:55:45.835651372 +0000 UTC m=+2991.001392647"
Sep 17 19:55:46.816289 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:46.816251    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:55:51.820633 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:51.820588    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:55:51.821197 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:51.821168    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:55:57.167373 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:57.167346    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:55:57.168728 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:55:57.168709    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 19:56:01.821087 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:01.821050    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:02.102878 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:02.102806    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:56:07.245898 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:07.245867    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:56:07.532336 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:07.532310    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:56:11.821460 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:11.821422    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:21.821638 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:21.821575    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:31.822031 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:31.821993    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:41.821680 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:41.821636    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:51.821808 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:51.821771    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:56:54.163726 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:54.163685    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:56:54.214381 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:56:54.214347    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:57:00.010512 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:00.010462    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Sep 17 19:57:10.011824 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:10.011794    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:57:13.303907 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:13.303879    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:57:13.304365 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:13.304109    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container" containerID="cri-o://a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552" gracePeriod=30
Sep 17 19:57:13.304954 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:13.304462    2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kube-rbac-proxy" containerID="cri-o://8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923" gracePeriod=30
Sep 17 19:57:13.327592 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:13.327569    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:57:14.184951 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:14.184917    2586 generic.go:356] "Generic (PLEG): container finished" podID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerID="8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923" exitCode=2
Sep 17 19:57:14.185127 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:14.184992    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerDied","Data":"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"}
Sep 17 19:57:16.816722 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:16.816679    2586 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Sep 17 19:57:16.944870 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:16.944847    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:57:17.095094 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095012    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert\") pod \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") "
Sep 17 19:57:17.095094 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095051    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcf4p\" (UniqueName: \"kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p\") pod \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") "
Sep 17 19:57:17.095301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095100    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location\") pod \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") "
Sep 17 19:57:17.095301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095157    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") pod \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") "
Sep 17 19:57:17.095301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095236    2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config\") pod \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\" (UID: \"4814cb8d-e65c-42bd-8762-64fb07ae87e5\") "
Sep 17 19:57:17.095467 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095414    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4814cb8d-e65c-42bd-8762-64fb07ae87e5" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 19:57:17.095467 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095422    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "4814cb8d-e65c-42bd-8762-64fb07ae87e5" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:57:17.095579 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095481    2586 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kserve-provision-location\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:57:17.095579 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095499    2586 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-cabundle-cert\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:57:17.095731 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.095692    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4814cb8d-e65c-42bd-8762-64fb07ae87e5" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 19:57:17.097419 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.097391    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4814cb8d-e65c-42bd-8762-64fb07ae87e5" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 19:57:17.097528 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.097436    2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p" (OuterVolumeSpecName: "kube-api-access-kcf4p") pod "4814cb8d-e65c-42bd-8762-64fb07ae87e5" (UID: "4814cb8d-e65c-42bd-8762-64fb07ae87e5"). InnerVolumeSpecName "kube-api-access-kcf4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 19:57:17.195908 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.195876    2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcf4p\" (UniqueName: \"kubernetes.io/projected/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-api-access-kcf4p\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:57:17.195908 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.195904    2586 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4814cb8d-e65c-42bd-8762-64fb07ae87e5-proxy-tls\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:57:17.195908 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.195915    2586 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4814cb8d-e65c-42bd-8762-64fb07ae87e5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-45.ec2.internal\" DevicePath \"\""
Sep 17 19:57:17.200044 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.200017    2586 generic.go:356] "Generic (PLEG): container finished" podID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerID="a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552" exitCode=0
Sep 17 19:57:17.200156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.200074    2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"
Sep 17 19:57:17.200156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.200092    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerDied","Data":"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"}
Sep 17 19:57:17.200156 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.200151    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd" event={"ID":"4814cb8d-e65c-42bd-8762-64fb07ae87e5","Type":"ContainerDied","Data":"11bc324d60e3b9da3841345e274cd005403888a58f5f2f97d6da7a1809e004ce"}
Sep 17 19:57:17.200265 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.200177    2586 scope.go:117] "RemoveContainer" containerID="8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"
Sep 17 19:57:17.208901 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.208884    2586 scope.go:117] "RemoveContainer" containerID="a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"
Sep 17 19:57:17.217030 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.217013    2586 scope.go:117] "RemoveContainer" containerID="8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"
Sep 17 19:57:17.223386 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.223365    2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:57:17.224768 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.224753    2586 scope.go:117] "RemoveContainer" containerID="8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"
Sep 17 19:57:17.225048 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:57:17.225026    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923\": container with ID starting with 8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923 not found: ID does not exist" containerID="8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"
Sep 17 19:57:17.225196 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.225051    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923"} err="failed to get container status \"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923\": rpc error: code = NotFound desc = could not find container \"8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923\": container with ID starting with 8092c6b18d49584ff6ef1a47201b0f592fcb524b0c002566fbb88d6590b6e923 not found: ID does not exist"
Sep 17 19:57:17.225196 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.225073    2586 scope.go:117] "RemoveContainer" containerID="a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"
Sep 17 19:57:17.225512 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:57:17.225481    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552\": container with ID starting with a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552 not found: ID does not exist" containerID="a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"
Sep 17 19:57:17.225719 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.225514    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552"} err="failed to get container status \"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552\": rpc error: code = NotFound desc = could not find container \"a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552\": container with ID starting with a8061e25d45dc872271d152aec16fc4ac8f3d9fc3468d4c37e594c800b567552 not found: ID does not exist"
Sep 17 19:57:17.225719 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.225537    2586 scope.go:117] "RemoveContainer" containerID="8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"
Sep 17 19:57:17.226029 ip-10-0-134-45 kubenswrapper[2586]: E0917 19:57:17.226005    2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe\": container with ID starting with 8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe not found: ID does not exist" containerID="8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"
Sep 17 19:57:17.226113 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.226036    2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe"} err="failed to get container status \"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe\": rpc error: code = NotFound desc = could not find container \"8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe\": container with ID starting with 8e184963b52890bbb6423965202c84120a71c6f64240d845c5591d14f40316fe not found: ID does not exist"
Sep 17 19:57:17.227333 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:17.227312    2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-p5ftd"]
Sep 17 19:57:19.015301 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:19.015265    2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" path="/var/lib/kubelet/pods/4814cb8d-e65c-42bd-8762-64fb07ae87e5/volumes"
Sep 17 19:57:48.993867 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:48.993836    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:57:54.122363 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:54.122323    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:57:54.753128 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:57:54.753090    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:58:02.643995 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:58:02.643956    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:59:39.367875 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:59:39.367796    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:59:39.731414 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:59:39.731333    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:59:55.421871 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:59:55.421832    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 19:59:55.476090 ip-10-0-134-45 kubenswrapper[2586]: I0917 19:59:55.476056    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:01.981072 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:01.981039    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:02.096590 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:02.096558    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:40.010808 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:40.010764    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:40.111458 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:40.111434    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:42.309875 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:42.309847    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vnlqc_dd1f444b-0e61-471b-9a77-6db913e39d7d/global-pull-secret-syncer/0.log"
Sep 17 20:00:42.399113 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:42.399089    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log"
Sep 17 20:00:42.538366 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:42.538341    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-45.ec2.internal_bc7c99acaec5e04bdd3d6997df869ed9/haproxy/0.log"
Sep 17 20:00:44.882998 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:44.882971    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:44.983460 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:44.983430    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:47.293682 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.293656    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8577cd5c85-m5rqn_dc0ce4fe-2e74-45ba-893e-5bbcd7e567ac/metrics-server/0.log"
Sep 17 20:00:47.358447 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.358420    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tthb_332b25c0-1317-475c-8bc2-e5dacfee288b/node-exporter/0.log"
Sep 17 20:00:47.380531 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.380514    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tthb_332b25c0-1317-475c-8bc2-e5dacfee288b/kube-rbac-proxy/0.log"
Sep 17 20:00:47.405463 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.405445    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6tthb_332b25c0-1317-475c-8bc2-e5dacfee288b/init-textfile/0.log"
Sep 17 20:00:47.934089 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.934064    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-5c6747dc7b-qksgz_f85a0321-e71f-4edf-af37-18cffea92d4b/prometheus-operator-admission-webhook/0.log"
Sep 17 20:00:47.964895 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.964866    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f6986b-dbx6l_c75ecd60-569f-499c-b1bc-8e3ca2e4632c/telemeter-client/0.log"
Sep 17 20:00:47.997913 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:47.997895    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f6986b-dbx6l_c75ecd60-569f-499c-b1bc-8e3ca2e4632c/reload/0.log"
Sep 17 20:00:48.030677 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:48.030656    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-655f6986b-dbx6l_c75ecd60-569f-499c-b1bc-8e3ca2e4632c/kube-rbac-proxy/0.log"
Sep 17 20:00:48.716842 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:48.716809    2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hnldj_7fdb8d8e-7afe-467e-aa92-694eb841f26d/konnectivity-agent/0.log" limit=1024
Sep 17 20:00:50.365844 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.365816    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6445cdb596-2k996_6468cb42-b2d1-496e-8c72-4f968a63b948/console/0.log"
Sep 17 20:00:50.716294 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716220    2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"]
Sep 17 20:00:50.716523 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716512    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kube-rbac-proxy"
Sep 17 20:00:50.716566 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716524    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kube-rbac-proxy"
Sep 17 20:00:50.716566 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716540    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="storage-initializer"
Sep 17 20:00:50.716566 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716547    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="storage-initializer"
Sep 17 20:00:50.716566 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716555    2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container"
Sep 17 20:00:50.716566 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716562    2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container"
Sep 17 20:00:50.716752 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716612    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kube-rbac-proxy"
Sep 17 20:00:50.716752 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.716634    2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4814cb8d-e65c-42bd-8762-64fb07ae87e5" containerName="kserve-container"
Sep 17 20:00:50.719501 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.719481    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.722970 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.722947    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdtgx\"/\"openshift-service-ca.crt\""
Sep 17 20:00:50.723101 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.722978    2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rdtgx\"/\"kube-root-ca.crt\""
Sep 17 20:00:50.723611 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.723596    2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rdtgx\"/\"default-dockercfg-bqm8w\""
Sep 17 20:00:50.731157 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.731138    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"]
Sep 17 20:00:50.764108 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.764085    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkdfl\" (UniqueName: \"kubernetes.io/projected/93805790-6d23-4d88-840d-19602344790f-kube-api-access-vkdfl\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.764230 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.764127    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-proc\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.764230 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.764146    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-lib-modules\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.764230 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.764161    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-podres\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.764368 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.764245    2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-sys\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865567 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865539    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-sys\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865757 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865582    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkdfl\" (UniqueName: \"kubernetes.io/projected/93805790-6d23-4d88-840d-19602344790f-kube-api-access-vkdfl\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865757 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865672    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-proc\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865757 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865690    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-sys\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865757 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865701    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-lib-modules\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865757 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865722    2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-podres\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865999 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865758    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-proc\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865999 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865837    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-lib-modules\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.865999 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.865855    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/93805790-6d23-4d88-840d-19602344790f-podres\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:50.873946 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:50.873926    2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkdfl\" (UniqueName: \"kubernetes.io/projected/93805790-6d23-4d88-840d-19602344790f-kube-api-access-vkdfl\") pod \"perf-node-gather-daemonset-9kr25\" (UID: \"93805790-6d23-4d88-840d-19602344790f\") " pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:51.029187 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:51.029167    2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:51.300672 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:51.300648    2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"]
Sep 17 20:00:51.303271 ip-10-0-134-45 kubenswrapper[2586]: W0917 20:00:51.303241    2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod93805790_6d23_4d88_840d_19602344790f.slice/crio-dd64a19ff8377e20a805569a48a2a7b61f42c872746bd46693d90a094cc77e1c WatchSource:0}: Error finding container dd64a19ff8377e20a805569a48a2a7b61f42c872746bd46693d90a094cc77e1c: Status 404 returned error can't find the container with id dd64a19ff8377e20a805569a48a2a7b61f42c872746bd46693d90a094cc77e1c
Sep 17 20:00:52.020367 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.020333    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrgzz_a071f8b2-95a9-4f3d-a1f5-503b42fd3f59/dns/0.log"
Sep 17 20:00:52.041227 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.041203    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zrgzz_a071f8b2-95a9-4f3d-a1f5-503b42fd3f59/kube-rbac-proxy/0.log"
Sep 17 20:00:52.064863 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.064829    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9kxdj_0a719325-0a92-413d-85d2-8848c0c0178f/dns-node-resolver/0.log"
Sep 17 20:00:52.087990 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.087963    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25" event={"ID":"93805790-6d23-4d88-840d-19602344790f","Type":"ContainerStarted","Data":"3212ee1091596f60545013c45e3723a18c75b2e962af88462222b35740381614"}
Sep 17 20:00:52.088117 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.088101    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:00:52.088185 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.088137    2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25" event={"ID":"93805790-6d23-4d88-840d-19602344790f","Type":"ContainerStarted","Data":"dd64a19ff8377e20a805569a48a2a7b61f42c872746bd46693d90a094cc77e1c"}
Sep 17 20:00:52.104160 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.104118    2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25" podStartSLOduration=2.104104979 podStartE2EDuration="2.104104979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 20:00:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 20:00:52.102605487 +0000 UTC m=+3297.268346781" watchObservedRunningTime="2026-09-17 20:00:52.104104979 +0000 UTC m=+3297.269846253"
Sep 17 20:00:52.576506 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.576478    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-74c559dfd4-pcvpj_90fcbb05-90ab-44ec-b698-89859beb174f/registry/0.log"
Sep 17 20:00:52.605262 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:52.605236    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2777q_0e054d15-f65f-4917-997c-85f1bf7d27eb/node-ca/0.log"
Sep 17 20:00:53.761101 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:53.761073    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-djkgc_6c1b263c-5248-4467-b56a-f11354ad6519/serve-healthcheck-canary/0.log"
Sep 17 20:00:54.166041 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:54.166013    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4lr2c_6594853e-0435-4597-9471-416da5138d56/kube-rbac-proxy/0.log"
Sep 17 20:00:54.189188 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:54.189167    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4lr2c_6594853e-0435-4597-9471-416da5138d56/exporter/0.log"
Sep 17 20:00:54.211515 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:54.211492    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4lr2c_6594853e-0435-4597-9471-416da5138d56/extractor/0.log"
Sep 17 20:00:56.701734 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:56.701699    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-66576f5db6-sgsxc_388856fa-d8f4-455b-8035-8e8a30d464d1/manager/0.log"
Sep 17 20:00:56.731167 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:56.731141    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_llmisvc-controller-manager-645f9c8fd6-46rp7_8183ffec-cca1-4146-bb29-2128a07e952f/manager/0.log"
Sep 17 20:00:56.751117 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:56.751096    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-nhfvm_032610a4-c869-474a-beff-9748c519be5f/server/0.log"
Sep 17 20:00:57.003673 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:57.003586    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-7b966d7984-cgwwt_05e5e562-7740-49e7-acdf-6e4bbfd6a9fd/manager/0.log"
Sep 17 20:00:57.192516 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:57.192480    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 20:00:57.194381 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:57.194360    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 20:00:57.226087 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:57.226059    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-custom-8b44d767d-d4lpg_29c7d402-e1ce-4396-9f26-6109635bbf20/seaweedfs-tls-custom/0.log"
Sep 17 20:00:57.249269 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:57.249246    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-d4kc5_719d144c-2374-4158-b31e-e24c5291ce49/seaweedfs-tls-serving/0.log"
Sep 17 20:00:58.102031 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:00:58.102002    2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-rdtgx/perf-node-gather-daemonset-9kr25"
Sep 17 20:01:02.925806 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:02.925777    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/kube-multus-additional-cni-plugins/0.log"
Sep 17 20:01:02.951885 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:02.951857    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/egress-router-binary-copy/0.log"
Sep 17 20:01:02.972138 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:02.972116    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/cni-plugins/0.log"
Sep 17 20:01:02.995560 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:02.995541    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/bond-cni-plugin/0.log"
Sep 17 20:01:03.018278 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.018258    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/routeoverride-cni/0.log"
Sep 17 20:01:03.041304 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.041282    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/whereabouts-cni-bincopy/0.log"
Sep 17 20:01:03.063252 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.063232    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-gqnlj_55913654-3b16-495f-894f-400ad02b2485/whereabouts-cni/0.log"
Sep 17 20:01:03.424496 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.424466    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n7f8p_dedcdb4b-2a47-461b-9b6a-f2067b550374/kube-multus/0.log"
Sep 17 20:01:03.564600 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.564571    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2gkt_b563966d-464c-4962-9c8e-71ba17af1708/network-metrics-daemon/0.log"
Sep 17 20:01:03.583365 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:03.583342    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-m2gkt_b563966d-464c-4962-9c8e-71ba17af1708/kube-rbac-proxy/0.log"
Sep 17 20:01:04.598911 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.598887    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-controller/0.log"
Sep 17 20:01:04.619150 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.619120    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/0.log"
Sep 17 20:01:04.619436 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.619406    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovn-acl-logging/1.log"
Sep 17 20:01:04.637227 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.637205    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/kube-rbac-proxy-node/0.log"
Sep 17 20:01:04.661749 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.661727    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/kube-rbac-proxy-ovn-metrics/0.log"
Sep 17 20:01:04.686380 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.686359    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/northd/0.log"
Sep 17 20:01:04.710379 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.710359    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/nbdb/0.log"
Sep 17 20:01:04.735702 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.735679    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/sbdb/0.log"
Sep 17 20:01:04.778087 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:04.778066    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6p2vh_6b359f42-0dbc-4595-bf96-c67f951fb099/ovnkube-controller/0.log"
Sep 17 20:01:06.094505 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:06.094433    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-m86xm_3d798031-06ea-48a0-90e7-f4c6928307d1/network-check-target-container/0.log"
Sep 17 20:01:07.092651 ip-10-0-134-45 kubenswrapper[2586]: I0917 20:01:07.092609    2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8f4nq_102b7be6-ec29-407d-bb39-83b4f00acefb/iptables-alerter/0.log"