Jul 28 01:23:44.398637 ip-10-0-134-120 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 01:23:44.398646 ip-10-0-134-120 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 01:23:44.398653 ip-10-0-134-120 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 01:23:44.398880 ip-10-0-134-120 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 01:23:54.439239 ip-10-0-134-120 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 01:23:54.439258 ip-10-0-134-120 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cd4cfee86193475d8a6cd24f481c0cc7 --
Jul 28 01:25:50.975563 ip-10-0-134-120 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 01:25:51.351682 ip-10-0-134-120 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:51.351682 ip-10-0-134-120 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 01:25:51.351682 ip-10-0-134-120 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:51.351682 ip-10-0-134-120 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 01:25:51.351682 ip-10-0-134-120 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 01:25:51.353338 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.353202    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 01:25:51.358803 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358769    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358806    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358812    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358815    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358819    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358822    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358825    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358828    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358831    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358835    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358838    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358841    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358844    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358847    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358849    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358852    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358855    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358858    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358861    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358864    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.358858 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358867    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358870    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358873    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358876    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358878    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358881    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358885    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358889    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358892    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358894    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358903    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358905    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358909    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358911    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358914    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358917    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358919    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358922    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358924    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.359338 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358927    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358929    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358932    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358935    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358939    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358943    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358946    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358948    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358951    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358954    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358957    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358961    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358964    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358968    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358971    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358974    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358977    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358981    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358984    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358986    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.359865 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358989    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358992    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358994    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.358997    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359000    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359003    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359006    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359008    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359011    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359014    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359016    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359019    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359022    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359024    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359030    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359034    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359037    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359040    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359044    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.360367 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359049    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359051    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359054    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359057    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359060    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359062    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359065    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359068    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359071    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359074    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359076    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359079    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359082    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359085    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359087    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359090    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359092    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359095    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359098    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359100    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.360851 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359103    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359913    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359922    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359926    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359930    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359933    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359937    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359941    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359945    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359947    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359950    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359954    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359957    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359961    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359964    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359967    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359970    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359973    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359975    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.361349 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359978    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359982    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359985    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359987    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359990    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359995    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.359998    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360000    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360004    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360006    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360009    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360011    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360014    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360017    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360020    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360022    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360026    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360030    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360033    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.361855 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360036    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360039    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360042    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360045    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360048    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360051    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360054    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360058    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360060    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360063    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360066    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360068    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360072    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360075    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360078    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360080    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360083    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360086    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360088    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360091    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.362327 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360094    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360096    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360099    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360101    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360104    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360106    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360109    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360111    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360114    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360117    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360120    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360122    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360125    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360128    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360130    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360133    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360135    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360139    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360142    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360145    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.362981 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360148    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360151    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360153    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360156    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360160    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360163    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360166    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360169    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360171    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360174    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360177    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360179    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360182    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360185    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360187    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360190    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360193    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360196    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360199    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360201    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.363477 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360204    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360206    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360639    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360646    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360649    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360653    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360656    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360659    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360662    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360665    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360667    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360671    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360673    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360676    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360679    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360681    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360685    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360688    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360690    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360693    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.364002 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360695    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360698    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360701    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360704    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360707    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360710    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360713    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360716    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360718    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360721    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360723    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360725    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360728    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360730    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360733    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360736    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360739    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360742    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360744    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360747    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.364497 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360749    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360752    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360755    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360757    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360760    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360762    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360765    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360770    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360773    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360775    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360778    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360780    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360783    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360786    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360788    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360791    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360812    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360817    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360821    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.364999 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360825    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360829    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360833    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360835    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360838    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360840    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360843    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360846    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360853    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360855    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360858    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360861    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360865    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360868    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360871    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360874    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360876    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360879    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360882    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360884    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.365473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360888    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360891    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360894    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360896    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360899    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360902    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360904    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360908    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360911    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360915    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360918    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360922    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360925    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360927    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360931    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360935    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360938    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360941    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360945    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.365975 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360949    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360953    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.360956    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361036    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361043    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361050    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361054    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361059    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361062    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361067    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361071    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361075    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361078    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361083    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361086    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361090    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361093    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361096    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361099    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361102    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361105    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361110    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361113    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361116    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 01:25:51.366452 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361119    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361123    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361128    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361131    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361134    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361138    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361141    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361144    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361147    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361151    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361154    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361162    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361165    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361168    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361171    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361174    2581 flags.go:64] FLAG: --enable-server="true"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361177    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361182    2581 flags.go:64] FLAG: --event-burst="100"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361185    2581 flags.go:64] FLAG: --event-qps="50"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361188    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361191    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361195    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361199    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361202    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361205    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 01:25:51.367047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361208    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361211    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361214    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361217    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361220    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361223    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361226    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361229    2581 flags.go:64] FLAG: --feature-gates=""
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361233    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361238    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361241    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361244    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361247    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361251    2581 flags.go:64] FLAG: --help="false"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361254    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361258    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361262    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361274    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361278    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361282    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361285    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361288    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361291    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 01:25:51.367666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361294    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361297    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361300    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361303    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361306    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361310    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361314    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361317    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361320    2581 flags.go:64] FLAG: --lock-file=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361322    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361325    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361328    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361334    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361336    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361339    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361342    2581 flags.go:64] FLAG: --logging-format="text"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361345    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361349    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361351    2581 flags.go:64] FLAG: --manifest-url=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361355    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361360    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361364    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361369    2581 flags.go:64] FLAG: --max-pods="110"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361372    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361376    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 01:25:51.368341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361380    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361383    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361386    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361389    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361392    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361400    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361403    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361406    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361409    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361412    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361418    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361421    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361425    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361428    2581 flags.go:64] FLAG: --port="10250"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361431    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361434    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0e769038d271d6622"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361437    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361440    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361443    2581 flags.go:64] FLAG: --register-node="true"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361445    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361449    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361452    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361455    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361458    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361462    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 01:25:51.369401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361465    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361468    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361471    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361474    2581 flags.go:64] FLAG: --runonce="false"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361477    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361480    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361483    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361486    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361489    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361492    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361496    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361499    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361502    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361505    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361508    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361511    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361515    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361518    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361521    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361527    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361530    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361533    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361538    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361541    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361544    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 01:25:51.370079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361547    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361550    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361554    2581 flags.go:64] FLAG: --v="2"
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361558    2581 flags.go:64] FLAG: --version="false"
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361562    2581 flags.go:64] FLAG: --vmodule=""
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361566    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.361570    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361690    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361694    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361697    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361706    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361709    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361711    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361714    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361717    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361720    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361723    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361725    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361728    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361730    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361733    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.370730 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361735    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361738    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361740    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361742    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361745    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361748    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361751    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361753    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361756    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361759    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361761    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361764    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361766    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361769    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361772    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361774    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361776    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361779    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361782    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361785    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.371288 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361787    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361790    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361806    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361809    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361811    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361814    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361817    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361820    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361822    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361825    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361828    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361830    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361833    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361835    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361838    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361840    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361843    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361846    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361848    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361851    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.371807 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361855    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361859    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361862    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361865    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361867    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361870    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361873    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361875    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361878    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361882    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361885    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361887    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361890    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361893    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361897    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361902    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361905    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361908    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361911    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.372388 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361913    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361916    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361918    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361921    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361924    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361926    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361929    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361932    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361935    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361938    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361942    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361945    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361947    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361950    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361952    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361955    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361957    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361960    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361963    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361966    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.372901 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361968    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361971    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361976    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361978    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361981    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.361985    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.362661    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.369657    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.369679    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.370350    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.370359    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370428    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370435    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370438    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.373642 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370442    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370445    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370449    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370452    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370455    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370458    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370462    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370464    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370467    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370472    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370477    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370481    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370484    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370487    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370489    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370492    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370495    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370498    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370500    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370503    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.374240 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370506    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370509    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370511    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370514    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370517    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370520    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370524    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370528    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370531    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370535    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370537    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370540    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370543    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370546    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370548    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370553    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370557    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370560    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370563    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.374960 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370566    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370569    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370572    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370575    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370578    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370580    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370583    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370586    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370588    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370591    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370594    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370597    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370599    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370603    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370606    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370609    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370611    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370614    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370617    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370620    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.375595 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370623    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370627    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370630    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370633    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370635    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370638    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370641    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370643    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370646    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370649    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370652    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370654    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370657    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370660    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370662    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370665    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370668    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370671    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370673    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370694    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.376221 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370698    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370701    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370705    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370707    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370711    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370714    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370716    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370719    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370722    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370726    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370729    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370732    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370735    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370740    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370744    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370748    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.376751 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370751    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.370757    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370894    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370899    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370902    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370905    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370908    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370911    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370914    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370917    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370920    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370923    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370927    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370931    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370934    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 01:25:51.377220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370937    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370940    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370943    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370946    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370949    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370951    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370954    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370958    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370961    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370964    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370967    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370970    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370974    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370978    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370981    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370984    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370988    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370990    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370993    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370996    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 01:25:51.377627 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.370999    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371002    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371004    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371007    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371010    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371013    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371016    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371018    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371021    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371024    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371026    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371029    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371032    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371035    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371038    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371040    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371043    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371046    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371048    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371051    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 01:25:51.378205 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371054    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371057    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371060    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371062    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371065    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371068    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371070    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371073    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371076    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371083    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371086    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371089    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371092    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371095    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371097    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371100    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371103    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371105    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371108    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 01:25:51.378724 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371110    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371113    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371117    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371119    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371122    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371125    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371127    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371130    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371133    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371136    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371138    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371141    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371144    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371147    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371155    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371158    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371160    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371163    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371166    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371168    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 01:25:51.379220 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371171    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371174    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371177    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371180    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371184    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371188    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:51.371191    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.371197    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.371340    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.373723    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.375533    2581 server.go:1019] "Starting client certificate rotation"
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.375649    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:51.379759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.375691    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 01:25:51.380154 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.378046    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:51.382580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.382550    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 01:25:51.396474 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.396444    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 28 01:25:51.398780 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.398758    2581 log.go:25] "Validated CRI v1 image API"
Jul 28 01:25:51.399827 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.399790    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 01:25:51.403511 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.403480    2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 bae7b672-9936-4f4c-a709-8828dd243ffc:/dev/nvme0n1p4 d965c4a6-0be8-4216-9347-dd8864175766:/dev/nvme0n1p3]
Jul 28 01:25:51.403613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.403509    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 01:25:51.408175 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.408157    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:51.408741 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.408619    2581 manager.go:216] Machine: {Timestamp:2026-07-28 01:25:51.40809374 +0000 UTC m=+0.313663131 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101795 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ddaff05643a51ff79fce88c58a8c1 SystemUUID:ec2ddaff-0564-3a51-ff79-fce88c58a8c1 BootID:cd4cfee8-6193-475d-8a6c-d24f481c0cc7 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f2:e9:07:ba:dd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f2:e9:07:ba:dd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:8e:81:24:e9:8b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 01:25:51.408741 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.408738    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 01:25:51.408862 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.408830    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 01:25:51.411829 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.411804    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 01:25:51.412031 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.411846    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-120.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 01:25:51.413121 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.413109    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 01:25:51.413166 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.413125    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 01:25:51.413166 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.413134    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:51.413166 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.413145    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 01:25:51.414437 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.414425    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 01:25:51.417821 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.417806    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:51.418018 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418008    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 01:25:51.418053 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418039    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 01:25:51.418053 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418050    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 01:25:51.418109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418062    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 01:25:51.418109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418071    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 28 01:25:51.418109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418081    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 01:25:51.418694 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418679    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:51.418776 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.418705    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 01:25:51.422089 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.422049    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 01:25:51.423677 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.423654    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 01:25:51.423775 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.423692    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 01:25:51.424923 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424910    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424927    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424934    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424940    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424949    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424959    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424970    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424976    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 01:25:51.424982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424983    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 01:25:51.425204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424989    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 01:25:51.425204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.424997    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 01:25:51.425204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.425003    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 01:25:51.425655 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.425644    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 01:25:51.425698 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.425656    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 01:25:51.429412 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.429394    2581 server.go:1268] "Started kubelet"
Jul 28 01:25:51.429533 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.429499    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 01:25:51.429587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.429502    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 01:25:51.429652 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.429592    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 01:25:51.430422 ip-10-0-134-120 systemd[1]: Started Kubernetes Kubelet.
Jul 28 01:25:51.431049 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.430959    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 01:25:51.431049 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.430959    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 01:25:51.431179 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.431137    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-120.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 01:25:51.431330 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.431316    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 01:25:51.431542 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.431528    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 01:25:51.435408 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.434430    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-120.ec2.internal.18c64fb7674809a8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-120.ec2.internal,UID:ip-10-0-134-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-120.ec2.internal,},FirstTimestamp:2026-07-28 01:25:51.429364136 +0000 UTC m=+0.334933527,LastTimestamp:2026-07-28 01:25:51.429364136 +0000 UTC m=+0.334933527,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-120.ec2.internal,}"
Jul 28 01:25:51.436211 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.436192    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 01:25:51.437880 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.437855    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 01:25:51.437981 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.437875    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:51.438041 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.437997    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.438547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438477    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 01:25:51.438547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438504    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 01:25:51.438547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438477    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 01:25:51.438723 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438633    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 01:25:51.438723 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438646    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 01:25:51.438888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438866    2581 factory.go:55] Registering systemd factory
Jul 28 01:25:51.438967 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.438892    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 28 01:25:51.439131 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439114    2581 factory.go:153] Registering CRI-O factory
Jul 28 01:25:51.439131 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439132    2581 factory.go:223] Registration of the crio container factory successfully
Jul 28 01:25:51.439219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439181    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 01:25:51.439219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439201    2581 factory.go:103] Registering Raw factory
Jul 28 01:25:51.439219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439213    2581 manager.go:1196] Started watching for new ooms in manager
Jul 28 01:25:51.439624 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.439607    2581 manager.go:349] Starting recovery of all containers
Jul 28 01:25:51.441998 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.441967    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-120.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 01:25:51.442098 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.442055    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 01:25:51.443690 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.443666    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xdkqc"
Jul 28 01:25:51.451451 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.451429    2581 manager.go:354] Recovery completed
Jul 28 01:25:51.455068 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.455032    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xdkqc"
Jul 28 01:25:51.456768 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.456749    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.458669 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.458650    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.458753 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.458680    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.458753 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.458713    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.459351 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.459333    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 01:25:51.459351 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.459348    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 01:25:51.459469 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.459367    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 01:25:51.459469 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.459373    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.460125 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.460110    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.460189 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.460138    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.460189 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.460149    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.461180 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.461165    2581 policy_none.go:49] "None policy: Start"
Jul 28 01:25:51.461240 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.461186    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 01:25:51.461240 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.461198    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 01:25:51.461399 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.461304    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-120.ec2.internal.18c64fb7690727fa  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-120.ec2.internal,UID:ip-10-0-134-120.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-120.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-120.ec2.internal,},FirstTimestamp:2026-07-28 01:25:51.45866649 +0000 UTC m=+0.364235882,LastTimestamp:2026-07-28 01:25:51.45866649 +0000 UTC m=+0.364235882,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-120.ec2.internal,}"
Jul 28 01:25:51.462551 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.462537    2581 policy_none.go:47] "Start"
Jul 28 01:25:51.501698 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.501505    2581 manager.go:337] "Starting Device Plugin manager"
Jul 28 01:25:51.501856 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.501724    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 01:25:51.501856 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.501738    2581 server.go:87] "Starting device plugin registration server"
Jul 28 01:25:51.502088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.502072    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 01:25:51.502149 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.502090    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 01:25:51.502252 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.502187    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 01:25:51.502324 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.502286    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 01:25:51.502324 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.502295    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 01:25:51.502961 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.502937    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 01:25:51.503072 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.502999    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.603169 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.603089    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.604318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.604087    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.604318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.604115    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.604318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.604130    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.604318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.604168    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.613109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.613078    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.613109 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.613112    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-120.ec2.internal\": node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.629238 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.629207    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.640465 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.640418    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 01:25:51.641704 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.641682    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 01:25:51.641825 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.641722    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 01:25:51.641825 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.641744    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 01:25:51.641825 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.641778    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 01:25:51.643916 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.643887    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:51.729439 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.729397    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.742317 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.742292    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"]
Jul 28 01:25:51.742388 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.742375    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.743951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.743929    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.744047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.743966    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.744047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.743982    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.745580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.745566    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.745715 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.745698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.745758 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.745734    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.746417 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746397    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.746530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746423    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.746530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746437    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.746530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746488    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.746530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746509    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.746530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.746521    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.747896 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.747882    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.747965 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.747906    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 01:25:51.748615 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.748597    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 01:25:51.748615 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.748618    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 01:25:51.748748 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.748628    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeHasSufficientPID"
Jul 28 01:25:51.769836 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.769798    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-120.ec2.internal\" not found" node="ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.775049 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.775031    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-120.ec2.internal\" not found" node="ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.829717 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.829675    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.841166 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.841136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.841308 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.841171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.841308 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.841198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f6d61b851655e512466d358f996bf98-config\") pod \"kube-apiserver-proxy-ip-10-0-134-120.ec2.internal\" (UID: \"3f6d61b851655e512466d358f996bf98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.930425 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:51.930355    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:51.941913 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.941881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.942016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.941888    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.942016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.941959    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.942016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.942006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f6d61b851655e512466d358f996bf98-config\") pod \"kube-apiserver-proxy-ip-10-0-134-120.ec2.internal\" (UID: \"3f6d61b851655e512466d358f996bf98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.942126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.942014    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3f6d61b851655e512466d358f996bf98-config\") pod \"kube-apiserver-proxy-ip-10-0-134-120.ec2.internal\" (UID: \"3f6d61b851655e512466d358f996bf98\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:51.942126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:51.942027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/add93d260e587bbdd5e4a8d2d39e4e8f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal\" (UID: \"add93d260e587bbdd5e4a8d2d39e4e8f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:52.030920 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.030875    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.073935 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.073908    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:52.079296 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.078983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:52.132070 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.132016    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.234315 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.234225    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.311700 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.311671    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:52.332266 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.332239    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:52.335186 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.335166    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.374714 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.374677    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 01:25:52.375187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.374896    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:52.375187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.374902    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:52.375187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.374903    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 01:25:52.435909 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.435878    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.457458 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.457410    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 01:20:51 +0000 UTC" deadline="2028-05-06 17:30:06.790321228 +0000 UTC"
Jul 28 01:25:52.457458 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.457447    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15568h4m14.33287688s"
Jul 28 01:25:52.536822 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.536731    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.637293 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.637256    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.735363 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:52.735333    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f6d61b851655e512466d358f996bf98.slice/crio-f7bdf466fe15833534651422795df9e0533238096bc9286c1b5bb7242d2e84a2 WatchSource:0}: Error finding container f7bdf466fe15833534651422795df9e0533238096bc9286c1b5bb7242d2e84a2: Status 404 returned error can't find the container with id f7bdf466fe15833534651422795df9e0533238096bc9286c1b5bb7242d2e84a2
Jul 28 01:25:52.735667 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:52.735649    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd93d260e587bbdd5e4a8d2d39e4e8f.slice/crio-6c04bb9567e9a89963c4a48d5d5293815e0b138a71e486e48d770d217728ee0b WatchSource:0}: Error finding container 6c04bb9567e9a89963c4a48d5d5293815e0b138a71e486e48d770d217728ee0b: Status 404 returned error can't find the container with id 6c04bb9567e9a89963c4a48d5d5293815e0b138a71e486e48d770d217728ee0b
Jul 28 01:25:52.737506 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.737451    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.740917 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.740891    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:25:52.837628 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.837531    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.938315 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:52.938282    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-120.ec2.internal\" not found"
Jul 28 01:25:52.970232 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:52.970205    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:53.038240 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.038212    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:53.048176 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.048143    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:53.049056 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.049040    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal"
Jul 28 01:25:53.058267 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.058242    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 01:25:53.419708 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.419649    2581 apiserver.go:52] "Watching apiserver"
Jul 28 01:25:53.434302 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.434267    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 01:25:53.434860 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.434827    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-h6hhd","openshift-multus/multus-additional-cni-plugins-mvbd5","openshift-network-diagnostics/network-check-target-g79hr","openshift-ovn-kubernetes/ovnkube-node-zkwqs","kube-system/konnectivity-agent-cq2hp","kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal","openshift-dns/node-resolver-lt4pp","openshift-image-registry/node-ca-wnrqq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal","openshift-multus/multus-vc4kz","openshift-multus/network-metrics-daemon-tr4zt","openshift-network-operator/iptables-alerter-g4rcm","kube-system/global-pull-secret-syncer-hsjxk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"]
Jul 28 01:25:53.436822 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.436784    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.438529 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.438490    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 01:25:53.438653 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.438579    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.438710 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.438657    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:25:53.440023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.439871    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gl8hm\""
Jul 28 01:25:53.440023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.439883    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.440023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.439912    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.440227 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.440185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:53.440284 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.440267    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:25:53.442480 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.442460    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.444440 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.444421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.445199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.445178    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 01:25:53.445289 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.445200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 01:25:53.445438 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.445417    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-69pcn\""
Jul 28 01:25:53.445621 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.445605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 01:25:53.446219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.446047    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 01:25:53.446219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.446133    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.448192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.446699    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.448192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.447310    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.449183 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.448424    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 01:25:53.449183 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.448847    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 01:25:53.449183 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449025    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qw7qp\""
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449749    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d03f91-4264-43d2-85ae-ca0ffef006a7-hosts-file\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449782    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d03f91-4264-43d2-85ae-ca0ffef006a7-tmp-dir\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfwlm\" (UniqueName: \"kubernetes.io/projected/25d03f91-4264-43d2-85ae-ca0ffef006a7-kube-api-access-jfwlm\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449850    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-kubelet-config\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-dbus\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.450015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.449938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:53.451777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.450592    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.451777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.450960    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.451777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.451119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f2hn7\""
Jul 28 01:25:53.451777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.451261    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 01:25:53.451777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.451594    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 01:25:53.452670 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.452477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.454449 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.454145    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.455817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.455636    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 01:25:53.456000 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.455979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.456076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.456029    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.456129 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.456076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.456235 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.456176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.456485 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.456467    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-djp57\""
Jul 28 01:25:53.457389 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.457371    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4v68r\""
Jul 28 01:25:53.457477 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.457403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.457969 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.457947    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.458840 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.458822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.459592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.459575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.459692 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.459644    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:25:53.461316 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.461295    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-h6b6k\""
Jul 28 01:25:53.461838 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.461790    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-88t4j\""
Jul 28 01:25:53.462045 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.462022    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 01:25:53.462209 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.462191    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 01:25:53.462648 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.462631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.464230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.464209    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 01:25:53.464320 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.464239    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 01:25:53.464478 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.464409    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.465255 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.465203    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 01:25:53.465896 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.465876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 01:25:53.466193 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.466176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p4czj\""
Jul 28 01:25:53.466250 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.466176    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 01:25:53.478577 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.478551    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn79d"
Jul 28 01:25:53.487233 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.487207    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-gn79d"
Jul 28 01:25:53.539965 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.539937    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 01:25:53.550408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550417    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-host\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-tuned\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-etc-kubernetes\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550507    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-bin\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-config\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-sys-fs\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.550564    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550572    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.550610 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-socket-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.550660    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:54.050621088 +0000 UTC m=+2.956190463 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-kubernetes\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9pqp\" (UniqueName: \"kubernetes.io/projected/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-kube-api-access-g9pqp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550769    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550818    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-netd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d03f91-4264-43d2-85ae-ca0ffef006a7-tmp-dir\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.550902    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79tq5\" (UniqueName: \"kubernetes.io/projected/582e892b-4304-42c6-a258-71cca1770646-kube-api-access-79tq5\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551015    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-var-lib-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7878005c-1573-4937-8d3a-98bf70391b9e-host\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-run\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551101    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551122    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-os-release\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551145    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-node-log\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18fe994-d6a9-4deb-a295-243b7d78334d-ovn-node-metrics-cert\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d03f91-4264-43d2-85ae-ca0ffef006a7-hosts-file\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551244    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfwlm\" (UniqueName: \"kubernetes.io/projected/25d03f91-4264-43d2-85ae-ca0ffef006a7-kube-api-access-jfwlm\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/25d03f91-4264-43d2-85ae-ca0ffef006a7-tmp-dir\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551294    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68877fd4-0c06-4342-85ea-efa14703ff7e-iptables-alerter-script\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-netns\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.551352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-hostroot\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.551809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.551809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-lib-modules\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.551809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551708    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.551809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551735    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.551809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-env-overrides\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-script-lib\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551882    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-conf\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/25d03f91-4264-43d2-85ae-ca0ffef006a7-hosts-file\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551964    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-systemd-units\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.551992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cn5\" (UniqueName: \"kubernetes.io/projected/7878005c-1573-4937-8d3a-98bf70391b9e-kube-api-access-g4cn5\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.552032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-systemd\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552070    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-k8s-cni-cncf-io\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-slash\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-system-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-socket-dir-parent\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552269    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-multus\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.552298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-multus-certs\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552324    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvnt\" (UniqueName: \"kubernetes.io/projected/c18fe994-d6a9-4deb-a295-243b7d78334d-kube-api-access-zxvnt\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552358    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-device-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552414    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-var-lib-kubelet\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552435    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-tmp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-kubelet\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552517    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552540    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-ovn\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-modprobe-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552591    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-sys\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.552614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-cnibin\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552639    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-kubelet\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-multus-daemon-config\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-etc-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmpqp\" (UniqueName: \"kubernetes.io/projected/0daa1347-ccbb-49de-b326-a340026b27ab-kube-api-access-rmpqp\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7878005c-1573-4937-8d3a-98bf70391b9e-serviceca\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552780    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-kubelet-config\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552819    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-dbus\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysconfig\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-kubelet-config\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552897    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-os-release\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-cnibin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552940    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-systemd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-registration-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.552977    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553010    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl8hh\" (UniqueName: \"kubernetes.io/projected/119d49ad-c20e-42b4-ba59-4fead814dd6e-kube-api-access-dl8hh\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.553076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553040    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-dbus\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68877fd4-0c06-4342-85ea-efa14703ff7e-host-slash\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553077    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-bin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-conf-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-netns\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553167    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xdn\" (UniqueName: \"kubernetes.io/projected/68877fd4-0c06-4342-85ea-efa14703ff7e-kube-api-access-w9xdn\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553190    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-cni-binary-copy\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl884\" (UniqueName: \"kubernetes.io/projected/329df437-fb66-4775-a16e-94b06ef0cf80-kube-api-access-fl884\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-log-socket\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/831129bd-28d1-453f-bd7c-732dbaf3ccf4-agent-certs\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/831129bd-28d1-453f-bd7c-732dbaf3ccf4-konnectivity-ca\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.553675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.553356    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.558704 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.558635    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:53.558704 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.558708    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:53.558940 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.558722    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:53.558940 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.558904    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 01:25:53.558940 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.558918    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:54.058895319 +0000 UTC m=+2.964464715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:53.562767 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.562741    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfwlm\" (UniqueName: \"kubernetes.io/projected/25d03f91-4264-43d2-85ae-ca0ffef006a7-kube-api-access-jfwlm\") pod \"node-resolver-lt4pp\" (UID: \"25d03f91-4264-43d2-85ae-ca0ffef006a7\") " pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.648714 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.648662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal" event={"ID":"3f6d61b851655e512466d358f996bf98","Type":"ContainerStarted","Data":"f7bdf466fe15833534651422795df9e0533238096bc9286c1b5bb7242d2e84a2"}
Jul 28 01:25:53.649683 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.649654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal" event={"ID":"add93d260e587bbdd5e4a8d2d39e4e8f","Type":"ContainerStarted","Data":"6c04bb9567e9a89963c4a48d5d5293815e0b138a71e486e48d770d217728ee0b"}
Jul 28 01:25:53.653887 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653849    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-sys\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653898    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-cnibin\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-kubelet\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-multus-daemon-config\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-etc-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmpqp\" (UniqueName: \"kubernetes.io/projected/0daa1347-ccbb-49de-b326-a340026b27ab-kube-api-access-rmpqp\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653965    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-sys\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.653987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7878005c-1573-4937-8d3a-98bf70391b9e-serviceca\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.654016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-kubelet\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654021    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysconfig\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-etc-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654043    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-cnibin\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-os-release\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysconfig\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-cnibin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-systemd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654133    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-registration-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654180    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl8hh\" (UniqueName: \"kubernetes.io/projected/119d49ad-c20e-42b4-ba59-4fead814dd6e-kube-api-access-dl8hh\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-systemd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-os-release\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654206    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68877fd4-0c06-4342-85ea-efa14703ff7e-host-slash\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654231    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-cnibin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654268    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-bin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/68877fd4-0c06-4342-85ea-efa14703ff7e-host-slash\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.654408 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654231    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-bin\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654318    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-registration-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-conf-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654348    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-conf-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-netns\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654292    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-etc-selinux\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-netns\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xdn\" (UniqueName: \"kubernetes.io/projected/68877fd4-0c06-4342-85ea-efa14703ff7e-kube-api-access-w9xdn\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654417    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-cni-binary-copy\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl884\" (UniqueName: \"kubernetes.io/projected/329df437-fb66-4775-a16e-94b06ef0cf80-kube-api-access-fl884\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654467    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-log-socket\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654488    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/831129bd-28d1-453f-bd7c-732dbaf3ccf4-agent-certs\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/831129bd-28d1-453f-bd7c-732dbaf3ccf4-konnectivity-ca\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-host\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654599    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-tuned\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654608    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7878005c-1573-4937-8d3a-98bf70391b9e-serviceca\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.655204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654610    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-multus-daemon-config\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654663    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654674    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-host\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-etc-kubernetes\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654727    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-log-socket\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-bin\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654764    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-config\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654779    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-sys-fs\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654841    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-bin\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654856    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-socket-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654879    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-etc-kubernetes\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-kubernetes\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9pqp\" (UniqueName: \"kubernetes.io/projected/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-kube-api-access-g9pqp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-kubernetes\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654930    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/582e892b-4304-42c6-a258-71cca1770646-cni-binary-copy\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656114 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654973    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-netd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.654986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-sys-fs\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79tq5\" (UniqueName: \"kubernetes.io/projected/582e892b-4304-42c6-a258-71cca1770646-kube-api-access-79tq5\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-var-lib-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655063    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7878005c-1573-4937-8d3a-98bf70391b9e-host\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655073    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-run\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/831129bd-28d1-453f-bd7c-732dbaf3ccf4-konnectivity-ca\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-run\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-var-lib-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-socket-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7878005c-1573-4937-8d3a-98bf70391b9e-host\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655263    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-run-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655272    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-os-release\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-node-log\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655301    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-cni-netd\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18fe994-d6a9-4deb-a295-243b7d78334d-ovn-node-metrics-cert\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.656980 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655354    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655377    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-os-release\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68877fd4-0c06-4342-85ea-efa14703ff7e-iptables-alerter-script\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-netns\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-hostroot\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655460    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-lib-modules\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655501    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-node-log\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655538    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-netns\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-hostroot\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655606    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-lib-modules\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655637    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.655650    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-env-overrides\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:53.655723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:54.155706111 +0000 UTC m=+3.061275501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:53.657855 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-script-lib\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-conf\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655809    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-systemd-units\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655833    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cn5\" (UniqueName: \"kubernetes.io/projected/7878005c-1573-4937-8d3a-98bf70391b9e-kube-api-access-g4cn5\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655858    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-systemd\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655884    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-k8s-cni-cncf-io\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-slash\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655932    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-system-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.655979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-socket-dir-parent\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-multus\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-multus-certs\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvnt\" (UniqueName: \"kubernetes.io/projected/c18fe994-d6a9-4deb-a295-243b7d78334d-kube-api-access-zxvnt\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656078    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-device-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-var-lib-kubelet\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656131    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-env-overrides\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-tmp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.658684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656230    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-kubelet\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656273    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-ovn\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-config\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-modprobe-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656446    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-modprobe-d\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656484    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-kubelet\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656520    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-openvswitch\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656554    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-run-ovn\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656777    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656818    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/119d49ad-c20e-42b4-ba59-4fead814dd6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656839    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-multus-certs\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656880    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/119d49ad-c20e-42b4-ba59-4fead814dd6e-system-cni-dir\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656911    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-system-cni-dir\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.659491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656944    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-multus-socket-dir-parent\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.656976    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-var-lib-cni-multus\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657197    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0daa1347-ccbb-49de-b326-a340026b27ab-device-dir\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657208    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-sysctl-conf\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657237    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-systemd-units\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-var-lib-kubelet\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/582e892b-4304-42c6-a258-71cca1770646-host-run-k8s-cni-cncf-io\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657317    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c18fe994-d6a9-4deb-a295-243b7d78334d-host-slash\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657327    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c18fe994-d6a9-4deb-a295-243b7d78334d-ovnkube-script-lib\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-systemd\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.657832    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/68877fd4-0c06-4342-85ea-efa14703ff7e-iptables-alerter-script\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.658676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-tmp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.658901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/831129bd-28d1-453f-bd7c-732dbaf3ccf4-agent-certs\") pod \"konnectivity-agent-cq2hp\" (UID: \"831129bd-28d1-453f-bd7c-732dbaf3ccf4\") " pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.659240    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-etc-tuned\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.660035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.659676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c18fe994-d6a9-4deb-a295-243b7d78334d-ovn-node-metrics-cert\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.665685 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.665656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmpqp\" (UniqueName: \"kubernetes.io/projected/0daa1347-ccbb-49de-b326-a340026b27ab-kube-api-access-rmpqp\") pod \"aws-ebs-csi-driver-node-nwmj9\" (UID: \"0daa1347-ccbb-49de-b326-a340026b27ab\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.666164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.666137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl8hh\" (UniqueName: \"kubernetes.io/projected/119d49ad-c20e-42b4-ba59-4fead814dd6e-kube-api-access-dl8hh\") pod \"multus-additional-cni-plugins-mvbd5\" (UID: \"119d49ad-c20e-42b4-ba59-4fead814dd6e\") " pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.666251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.666211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl884\" (UniqueName: \"kubernetes.io/projected/329df437-fb66-4775-a16e-94b06ef0cf80-kube-api-access-fl884\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:53.666251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.666221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79tq5\" (UniqueName: \"kubernetes.io/projected/582e892b-4304-42c6-a258-71cca1770646-kube-api-access-79tq5\") pod \"multus-vc4kz\" (UID: \"582e892b-4304-42c6-a258-71cca1770646\") " pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.666365 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.666249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9pqp\" (UniqueName: \"kubernetes.io/projected/6aa0e6e3-a03d-43c8-9bef-090e34f885d4-kube-api-access-g9pqp\") pod \"tuned-h6hhd\" (UID: \"6aa0e6e3-a03d-43c8-9bef-090e34f885d4\") " pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.666588 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.666568    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xdn\" (UniqueName: \"kubernetes.io/projected/68877fd4-0c06-4342-85ea-efa14703ff7e-kube-api-access-w9xdn\") pod \"iptables-alerter-g4rcm\" (UID: \"68877fd4-0c06-4342-85ea-efa14703ff7e\") " pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.667267 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.667252    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvnt\" (UniqueName: \"kubernetes.io/projected/c18fe994-d6a9-4deb-a295-243b7d78334d-kube-api-access-zxvnt\") pod \"ovnkube-node-zkwqs\" (UID: \"c18fe994-d6a9-4deb-a295-243b7d78334d\") " pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.667376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.667356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cn5\" (UniqueName: \"kubernetes.io/projected/7878005c-1573-4937-8d3a-98bf70391b9e-kube-api-access-g4cn5\") pod \"node-ca-wnrqq\" (UID: \"7878005c-1573-4937-8d3a-98bf70391b9e\") " pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.751051 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.750959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-lt4pp"
Jul 28 01:25:53.759392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.759369    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:25:53.775329 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.775296    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:25:53.781273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.781243    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9"
Jul 28 01:25:53.789213 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.789186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnrqq"
Jul 28 01:25:53.797154 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.797129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h6hhd"
Jul 28 01:25:53.805075 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.805049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mvbd5"
Jul 28 01:25:53.813007 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.812980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vc4kz"
Jul 28 01:25:53.819891 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.819867    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-g4rcm"
Jul 28 01:25:53.908876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:53.908846    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:54.059867 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.059775    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:54.059867 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.059863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:54.060067 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.059959    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:54.060067 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.059983    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:54.060067 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.059996    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:54.060067 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.060011    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:54.060067 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.060064    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.060044135 +0000 UTC m=+3.965613513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:54.060279 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.060084    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.060077347 +0000 UTC m=+3.965646723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:54.160460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.160427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:54.160620 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.160598    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:54.160685 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:54.160673    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:55.160653525 +0000 UTC m=+4.066222903 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:54.489940 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.489844    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:53 +0000 UTC" deadline="2028-02-01 15:01:55.692312402 +0000 UTC"
Jul 28 01:25:54.489940 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.489896    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13285h36m1.202424023s"
Jul 28 01:25:54.620467 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.620431    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18fe994_d6a9_4deb_a295_243b7d78334d.slice/crio-50aad51f69f3afee2fbfe06b6ad424445904d938f75907a9b847145786afb3f4 WatchSource:0}: Error finding container 50aad51f69f3afee2fbfe06b6ad424445904d938f75907a9b847145786afb3f4: Status 404 returned error can't find the container with id 50aad51f69f3afee2fbfe06b6ad424445904d938f75907a9b847145786afb3f4
Jul 28 01:25:54.626930 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.626893    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68877fd4_0c06_4342_85ea_efa14703ff7e.slice/crio-0dc53f5d88717e1ac79fe9c1eeb32f6d1ddf199f3be5a60f73537a6e4ec9446c WatchSource:0}: Error finding container 0dc53f5d88717e1ac79fe9c1eeb32f6d1ddf199f3be5a60f73537a6e4ec9446c: Status 404 returned error can't find the container with id 0dc53f5d88717e1ac79fe9c1eeb32f6d1ddf199f3be5a60f73537a6e4ec9446c
Jul 28 01:25:54.627661 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.627613    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa0e6e3_a03d_43c8_9bef_090e34f885d4.slice/crio-17fbbb967dfb7fa36caf227862e54665a55abb969dc0ac1b9dfbbc86e7f48e96 WatchSource:0}: Error finding container 17fbbb967dfb7fa36caf227862e54665a55abb969dc0ac1b9dfbbc86e7f48e96: Status 404 returned error can't find the container with id 17fbbb967dfb7fa36caf227862e54665a55abb969dc0ac1b9dfbbc86e7f48e96
Jul 28 01:25:54.628717 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.628691    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7878005c_1573_4937_8d3a_98bf70391b9e.slice/crio-694124d817c85685ebd7caaf998c3ff724525d8312e73279429cd6dbcd344be8 WatchSource:0}: Error finding container 694124d817c85685ebd7caaf998c3ff724525d8312e73279429cd6dbcd344be8: Status 404 returned error can't find the container with id 694124d817c85685ebd7caaf998c3ff724525d8312e73279429cd6dbcd344be8
Jul 28 01:25:54.629475 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.629407    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d03f91_4264_43d2_85ae_ca0ffef006a7.slice/crio-58f7cd49d20558988ccc89b82a289705d5ecaf63835beac0d3f8890da5a5776f WatchSource:0}: Error finding container 58f7cd49d20558988ccc89b82a289705d5ecaf63835beac0d3f8890da5a5776f: Status 404 returned error can't find the container with id 58f7cd49d20558988ccc89b82a289705d5ecaf63835beac0d3f8890da5a5776f
Jul 28 01:25:54.631705 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.631232    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582e892b_4304_42c6_a258_71cca1770646.slice/crio-9efd170bd5420826f868eb4dae6e258690caf52e3e2ea713a969d4c2dc2e8d41 WatchSource:0}: Error finding container 9efd170bd5420826f868eb4dae6e258690caf52e3e2ea713a969d4c2dc2e8d41: Status 404 returned error can't find the container with id 9efd170bd5420826f868eb4dae6e258690caf52e3e2ea713a969d4c2dc2e8d41
Jul 28 01:25:54.633768 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:25:54.633731    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831129bd_28d1_453f_bd7c_732dbaf3ccf4.slice/crio-e723c5b711c343b39416dde5152d58eb96e0274c8b2ffe5420d1c4d2de9d1312 WatchSource:0}: Error finding container e723c5b711c343b39416dde5152d58eb96e0274c8b2ffe5420d1c4d2de9d1312: Status 404 returned error can't find the container with id e723c5b711c343b39416dde5152d58eb96e0274c8b2ffe5420d1c4d2de9d1312
Jul 28 01:25:54.654280 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.654237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerStarted","Data":"782d9d7d729183a2b2b52c82b264624919a6d5f3aa33e9d9b24220e4d6bfe197"}
Jul 28 01:25:54.656726 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.656693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq2hp" event={"ID":"831129bd-28d1-453f-bd7c-732dbaf3ccf4","Type":"ContainerStarted","Data":"e723c5b711c343b39416dde5152d58eb96e0274c8b2ffe5420d1c4d2de9d1312"}
Jul 28 01:25:54.661128 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.661097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc4kz" event={"ID":"582e892b-4304-42c6-a258-71cca1770646","Type":"ContainerStarted","Data":"9efd170bd5420826f868eb4dae6e258690caf52e3e2ea713a969d4c2dc2e8d41"}
Jul 28 01:25:54.662243 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.662217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lt4pp" event={"ID":"25d03f91-4264-43d2-85ae-ca0ffef006a7","Type":"ContainerStarted","Data":"58f7cd49d20558988ccc89b82a289705d5ecaf63835beac0d3f8890da5a5776f"}
Jul 28 01:25:54.663455 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.663427    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnrqq" event={"ID":"7878005c-1573-4937-8d3a-98bf70391b9e","Type":"ContainerStarted","Data":"694124d817c85685ebd7caaf998c3ff724525d8312e73279429cd6dbcd344be8"}
Jul 28 01:25:54.664502 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.664480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6hhd" event={"ID":"6aa0e6e3-a03d-43c8-9bef-090e34f885d4","Type":"ContainerStarted","Data":"17fbbb967dfb7fa36caf227862e54665a55abb969dc0ac1b9dfbbc86e7f48e96"}
Jul 28 01:25:54.665785 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.665765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"50aad51f69f3afee2fbfe06b6ad424445904d938f75907a9b847145786afb3f4"}
Jul 28 01:25:54.668463 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.668430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9" event={"ID":"0daa1347-ccbb-49de-b326-a340026b27ab","Type":"ContainerStarted","Data":"de1c7c19492a6a17d202d7d0515f8ab11d8ece08e9018adc5f0a222f50cca0db"}
Jul 28 01:25:54.671727 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.671706    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4rcm" event={"ID":"68877fd4-0c06-4342-85ea-efa14703ff7e","Type":"ContainerStarted","Data":"0dc53f5d88717e1ac79fe9c1eeb32f6d1ddf199f3be5a60f73537a6e4ec9446c"}
Jul 28 01:25:54.747718 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.747409    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:54.800288 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:54.800258    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.067496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.067604    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.067833    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.067855    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.067869    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.067942    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.06792042 +0000 UTC m=+5.973489799 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.068348    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.068485 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.068396    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.068382412 +0000 UTC m=+5.973951791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:55.168700 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.168659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:55.168913 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.168847    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.168913 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.168912    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:25:57.168893772 +0000 UTC m=+6.074463149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:55.491136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.490968    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 01:20:53 +0000 UTC" deadline="2028-01-11 00:48:50.737458727 +0000 UTC"
Jul 28 01:25:55.491136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.491079    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12767h22m55.246385134s"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.642515    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.642559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.642679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.643123    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.643184    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:55.643287 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:55.643253    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:25:55.685718 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.685638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal" event={"ID":"3f6d61b851655e512466d358f996bf98","Type":"ContainerStarted","Data":"5fbcc0726204fb30c2ac17be7ab9bfb7ea3508765b6383961aa2150ba20ec83d"}
Jul 28 01:25:55.703157 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.702082    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-120.ec2.internal" podStartSLOduration=2.702058593 podStartE2EDuration="2.702058593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:55.70160568 +0000 UTC m=+4.607175080" watchObservedRunningTime="2026-07-28 01:25:55.702058593 +0000 UTC m=+4.607627997"
Jul 28 01:25:55.705404 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.704448    2581 generic.go:356] "Generic (PLEG): container finished" podID="add93d260e587bbdd5e4a8d2d39e4e8f" containerID="3a8b66fa0a0c9b0288ed9275bc330e67201e360f6b2230fb5167c05521b297cb" exitCode=0
Jul 28 01:25:55.705404 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:55.704497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal" event={"ID":"add93d260e587bbdd5e4a8d2d39e4e8f","Type":"ContainerDied","Data":"3a8b66fa0a0c9b0288ed9275bc330e67201e360f6b2230fb5167c05521b297cb"}
Jul 28 01:25:56.723970 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:56.723933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal" event={"ID":"add93d260e587bbdd5e4a8d2d39e4e8f","Type":"ContainerStarted","Data":"ee318d631bcc81561890efbf9f03a2feaddf56cbe4e86c486d40b854c4b9baf0"}
Jul 28 01:25:56.739564 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:56.739495    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-120.ec2.internal" podStartSLOduration=3.7394714049999997 podStartE2EDuration="3.739471405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:25:56.738871959 +0000 UTC m=+5.644441357" watchObservedRunningTime="2026-07-28 01:25:56.739471405 +0000 UTC m=+5.645040804"
Jul 28 01:25:57.083731 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.083593    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:57.083731 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.083720    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:57.083976 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.083906    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:57.083976 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.083973    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:01.08395373 +0000 UTC m=+9.989523127 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:25:57.084453 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.084425    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:25:57.084453 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.084456    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:25:57.084627 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.084469    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:57.084627 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.084518    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:01.084502212 +0000 UTC m=+9.990071589 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:25:57.184918 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.184878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:57.185168 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.185101    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:57.185257 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.185180    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:01.185150396 +0000 UTC m=+10.090719795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:25:57.642957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.642919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:57.643139 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.642966    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:57.643139 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.643061    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:25:57.643271 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:57.643146    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:57.643321 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.643262    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:25:57.643377 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:57.643338    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:59.642807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:59.642949    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:59.643104    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:59.643194    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:25:59.643241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:25:59.643346 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:25:59.643322    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:01.118515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.118471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.118542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118680    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118696    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118723    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118737    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118738    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:09.118721874 +0000 UTC m=+18.024291249 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:01.119053 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.118852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:09.118781637 +0000 UTC m=+18.024351013 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:01.219815 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.219768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:01.220005 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.219924    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:01.220005 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.219992    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:09.219972745 +0000 UTC m=+18.125542145 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.643790    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.643929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.644346    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.644441    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:01.644549    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:01.644680 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:01.644640    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:03.642672 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:03.642630    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:03.642672 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:03.642647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:03.643204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:03.642638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:03.643204 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:03.642764    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:03.643204 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:03.642910    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:03.643204 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:03.643006    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:05.642923 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:05.642892    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:05.642923 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:05.642911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:05.643279 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:05.642911    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:05.643279 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:05.642999    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:05.643279 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:05.643104    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:05.643279 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:05.643193    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:07.643029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:07.642998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:07.643501 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:07.642998    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:07.643501 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:07.643136    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:07.643501 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:07.643005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:07.643501 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:07.643198    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:07.643501 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:07.643275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:09.181631 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.181592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.181644    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181745    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181765    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181784    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181810    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:25.181814931 +0000 UTC m=+34.087384321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:09.182178 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.181860    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:25.181850729 +0000 UTC m=+34.087420115 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:09.282272 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.282227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:09.282432 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.282408    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:09.282501 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.282490    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:25.282469768 +0000 UTC m=+34.188039150 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:09.642399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.642306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:09.643017 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.642963    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:09.643017 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.643001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:09.643225 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:09.643069    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:09.643279 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.643238    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:09.643371 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:09.643349    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:11.642812 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:11.642757    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:11.643242 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:11.642898    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:11.643242 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:11.642937    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:11.643242 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:11.642946    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:11.643242 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:11.643051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:11.643242 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:11.643122    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:13.642106 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:13.642062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:13.642587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:13.642062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:13.642587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:13.642181    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:13.642587 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:13.642176    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:13.642587 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:13.642262    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:13.642587 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:13.642341    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:14.764231 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.763850    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="205ad2a30b4901114b0ecc673b8fd49c4ac575fd836cad2f07fd993c34c7f22f" exitCode=0
Jul 28 01:26:14.764939 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.763943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"205ad2a30b4901114b0ecc673b8fd49c4ac575fd836cad2f07fd993c34c7f22f"}
Jul 28 01:26:14.766303 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.766258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cq2hp" event={"ID":"831129bd-28d1-453f-bd7c-732dbaf3ccf4","Type":"ContainerStarted","Data":"b2748e464caaa7da91feb5afd32bff5787bdc7a88e19ee6c8ce6f3dd0dd249bb"}
Jul 28 01:26:14.767993 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.767967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vc4kz" event={"ID":"582e892b-4304-42c6-a258-71cca1770646","Type":"ContainerStarted","Data":"e986c8bbd092d07a451043b13a985406e50d525b6f9d9ecb7c4af27e0a85608d"}
Jul 28 01:26:14.769910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.769876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-lt4pp" event={"ID":"25d03f91-4264-43d2-85ae-ca0ffef006a7","Type":"ContainerStarted","Data":"f3fe12edac2620636096e987cc70f88dd7c7596a01697e237d8e8e98300d5c15"}
Jul 28 01:26:14.771640 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.771599    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnrqq" event={"ID":"7878005c-1573-4937-8d3a-98bf70391b9e","Type":"ContainerStarted","Data":"db815585f42082f577e3c5cce7f4fe3319cf1cc8666b401e176df047aa04eda2"}
Jul 28 01:26:14.773470 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.773444    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h6hhd" event={"ID":"6aa0e6e3-a03d-43c8-9bef-090e34f885d4","Type":"ContainerStarted","Data":"ca13d1c69daee15a7d2bd4e2a82586c1a880d2e8333f9e8ca30894908e382f1b"}
Jul 28 01:26:14.776944 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.776926    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:26:14.777290 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777265    2581 generic.go:356] "Generic (PLEG): container finished" podID="c18fe994-d6a9-4deb-a295-243b7d78334d" containerID="81aae5a416d11469c03fec3493e35f501d861bffef8b7e655b95ae2ed4aa861a" exitCode=1
Jul 28 01:26:14.777392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"4346cebc71f8fd9e6ba3dc930ca1e278f7166661265767f771ed18ca04ba8c31"}
Jul 28 01:26:14.777445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"a708a787f7d341aad089712c25dd3fe3e61f1c04fa6e9d26604025bb2550f6b8"}
Jul 28 01:26:14.777481 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"b13ef09c321e28bf0a7c36a0db63fc9796fde9da3f504788a88418115401aaac"}
Jul 28 01:26:14.777529 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777511    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"2e3ea0cf78c8e5ddc635fe35676640f32f79657400c888723134bd271c7cff3f"}
Jul 28 01:26:14.777642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerDied","Data":"81aae5a416d11469c03fec3493e35f501d861bffef8b7e655b95ae2ed4aa861a"}
Jul 28 01:26:14.777642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.777617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"6d3837f5200f2ddaf0cf37919d8f1e9fa13aaa84be167155a1cd69596aecf931"}
Jul 28 01:26:14.778912 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.778894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9" event={"ID":"0daa1347-ccbb-49de-b326-a340026b27ab","Type":"ContainerStarted","Data":"20321c6f445058831224647b162290ba6f505951d94e20d21522b4b1cf349f58"}
Jul 28 01:26:14.815964 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.815847    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cq2hp" podStartSLOduration=9.078221479 podStartE2EDuration="23.815823635s" totalImagesPullingTime="14.737602156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.815048716 +0000 UTC m=+23.720618115" watchObservedRunningTime="2026-07-28 01:26:14.815823635 +0000 UTC m=+23.721393026"
Jul 28 01:26:14.834694 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.834665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:14.837655 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.837612    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h6hhd" podStartSLOduration=4.717036096 podStartE2EDuration="23.837597553s" totalImagesPullingTime="19.120561457s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.837133009 +0000 UTC m=+23.742702406" watchObservedRunningTime="2026-07-28 01:26:14.837597553 +0000 UTC m=+23.743166951"
Jul 28 01:26:14.853467 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.853415    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-lt4pp" podStartSLOduration=4.735412216 podStartE2EDuration="23.853397247s" totalImagesPullingTime="19.117985031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.852938273 +0000 UTC m=+23.758507670" watchObservedRunningTime="2026-07-28 01:26:14.853397247 +0000 UTC m=+23.758966641"
Jul 28 01:26:14.872450 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.872400    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wnrqq" podStartSLOduration=4.755320183 podStartE2EDuration="23.872381451s" totalImagesPullingTime="19.117061268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.871783648 +0000 UTC m=+23.777353047" watchObservedRunningTime="2026-07-28 01:26:14.872381451 +0000 UTC m=+23.777950848"
Jul 28 01:26:14.900618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.900563    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vc4kz" podStartSLOduration=4.768602541 podStartE2EDuration="23.900546487s" totalImagesPullingTime="19.131943946s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:14.900244556 +0000 UTC m=+23.805813954" watchObservedRunningTime="2026-07-28 01:26:14.900546487 +0000 UTC m=+23.806115884"
Jul 28 01:26:14.928609 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:14.928580    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:15.181200 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.181175    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 01:26:15.216834 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.216774    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:15.446164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.446076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:15.515685 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.515529    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T01:26:15.181193393Z","UUID":"b43d65b0-a949-463a-86d7-0101b679e338","Handler":null,"Name":"","Endpoint":""}
Jul 28 01:26:15.518240 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.518216    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 01:26:15.518406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.518264    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 01:26:15.642529 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.642494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:15.642704 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.642494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:15.642704 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:15.642632    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:15.642704 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.642494    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:15.642940 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:15.642704    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:15.642940 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:15.642754    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:15.784730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.784691    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9" event={"ID":"0daa1347-ccbb-49de-b326-a340026b27ab","Type":"ContainerStarted","Data":"38cc59d3c1e853087d544c6d578b4300191af43bfa138ec7381c9ad0b72f4be1"}
Jul 28 01:26:15.786581 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:15.786513    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-g4rcm" event={"ID":"68877fd4-0c06-4342-85ea-efa14703ff7e","Type":"ContainerStarted","Data":"f2f87dbc2cf884de846e947aca43bd91f14caa761f8c6fe56f9314ce45774382"}
Jul 28 01:26:16.207727 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:16.207634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:16.645241 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:16.645192    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:16.792745 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:16.792490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9" event={"ID":"0daa1347-ccbb-49de-b326-a340026b27ab","Type":"ContainerStarted","Data":"136ae52531fc074f518d22fddb76e148f835ed31776110caa64f9e33632981f9"}
Jul 28 01:26:16.812963 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:16.812907    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-g4rcm" podStartSLOduration=6.738253055 podStartE2EDuration="25.812882089s" totalImagesPullingTime="19.074629034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:15.81165513 +0000 UTC m=+24.717224528" watchObservedRunningTime="2026-07-28 01:26:16.812882089 +0000 UTC m=+25.718451486"
Jul 28 01:26:16.817438 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:16.817408    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.021811 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.021737    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.219314 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.219285    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.497373 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.497301    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:17.606193 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.606162    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:26:17.606889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.606868    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:26:17.630395 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.630328    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nwmj9" podStartSLOduration=5.355275963 podStartE2EDuration="26.630302559s" totalImagesPullingTime="21.275026596s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:16.812788091 +0000 UTC m=+25.718357491" watchObservedRunningTime="2026-07-28 01:26:17.630302559 +0000 UTC m=+26.535871959"
Jul 28 01:26:17.642137 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.642100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:17.642355 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.642100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:17.642355 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.642100    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:17.642355 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:17.642260    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:17.642528 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:17.642372    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:17.642528 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:17.642452    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:17.801283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.801256    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:26:17.801866 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.801658    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"f268be97dcc77801d48ce272dfb12addf6808db19b85eee5f88773ade330d4e7"}
Jul 28 01:26:17.802132 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.802111    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:26:17.802586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.802569    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cq2hp"
Jul 28 01:26:17.805110 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:17.805085    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.046809 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:18.046745    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.417224 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:18.417187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.630137 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:18.630097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.831352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:18.831325    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:18.931306 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:18.931275    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.031679 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.031648    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.218360 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.218155    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.642390 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.642345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:19.642585 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.642345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:19.642585 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:19.642458    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:19.642585 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.642359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:19.642585 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:19.642543    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:19.642730 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:19.642648    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:19.808187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.808154    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="55415d8cd7bf9a83891652fbebfc034d608111b16670b58c0551c7ccc35c1f32" exitCode=0
Jul 28 01:26:19.808365 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.808242    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"55415d8cd7bf9a83891652fbebfc034d608111b16670b58c0551c7ccc35c1f32"}
Jul 28 01:26:19.810744 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.810716    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:19.811901 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.811880    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:26:19.812292 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.812269    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"0db466305ae4d67bf50752663d6c36118bcbc7f95aac24344fd5b69b1741eb3a"}
Jul 28 01:26:19.812515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.812494    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:19.812613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.812561    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:19.812613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.812610    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:19.812736 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.812717    2581 scope.go:117] "RemoveContainer" containerID="81aae5a416d11469c03fec3493e35f501d861bffef8b7e655b95ae2ed4aa861a"
Jul 28 01:26:19.830532 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.830510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:19.830643 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.830618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:19.960259 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:19.960173    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:20.060904 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:20.060869    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:20.438273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:20.438239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:20.817788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:20.817761    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:26:20.818107 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:20.818081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" event={"ID":"c18fe994-d6a9-4deb-a295-243b7d78334d","Type":"ContainerStarted","Data":"61e3865dd891b267e730551026efa3114f3d567f4093ff5a1455d78a76a3ec30"}
Jul 28 01:26:20.850845 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:20.850768    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs" podStartSLOduration=10.478843624 podStartE2EDuration="29.850752153s" totalImagesPullingTime="19.371908529s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:20.850443609 +0000 UTC m=+29.756013006" watchObservedRunningTime="2026-07-28 01:26:20.850752153 +0000 UTC m=+29.756321551"
Jul 28 01:26:21.210203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.210168    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.387862 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.387758    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.451545 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.451512    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.609515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.609483    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.643507 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.643440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:21.643613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.643603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:21.643662 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:21.643629    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:21.643706 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:21.643661    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:21.643742 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.643703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:21.643784 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:21.643772    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:21.808262 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.808226    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:21.822346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.822314    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="7cb1c1e198718883bf458bcfeddeedae4fbebc0bed777fd37dfea5b37b6afb4d" exitCode=0
Jul 28 01:26:21.822523 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:21.822383    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"7cb1c1e198718883bf458bcfeddeedae4fbebc0bed777fd37dfea5b37b6afb4d"}
Jul 28 01:26:22.016627 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:22.016596    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:23.229422 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.229328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:23.403978 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.403948    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:23.613669 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.613640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:23.642853 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.642826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:23.642853 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.642843    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:23.642983 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.642826    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:23.642983 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:23.642930    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:23.643050 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:23.643019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:23.643095 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:23.643078    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:23.828444 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.828412    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="7869bce688d519bbccde44357baed981afa9127f94a362f7aee4b6bc65e0dc55" exitCode=0
Jul 28 01:26:23.828606 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.828457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"7869bce688d519bbccde44357baed981afa9127f94a362f7aee4b6bc65e0dc55"}
Jul 28 01:26:23.833134 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:23.833098    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.018650 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:24.018618    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.134239 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:24.134208    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.611422 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:24.611387    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:24.866540 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:24.866452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:25.199409 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.199320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:25.199576 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.199484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:25.199576 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199490    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:25.199576 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199558    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret podName:9d8f5494-05fc-48a7-a8c2-cbfbf30134fb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:57.199538475 +0000 UTC m=+66.105107851 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret") pod "global-pull-secret-syncer-hsjxk" (UID: "9d8f5494-05fc-48a7-a8c2-cbfbf30134fb") : object "kube-system"/"original-pull-secret" not registered
Jul 28 01:26:25.199747 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199620    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 01:26:25.199747 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199639    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 01:26:25.199747 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199654    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xfsfb for pod openshift-network-diagnostics/network-check-target-g79hr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:25.199747 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.199709    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb podName:c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:57.199694039 +0000 UTC m=+66.105263418 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xfsfb" (UniqueName: "kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb") pod "network-check-target-g79hr" (UID: "c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 01:26:25.300760 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.300724    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:25.300976 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.300867    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:25.300976 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.300939    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs podName:329df437-fb66-4775-a16e-94b06ef0cf80 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:57.300922301 +0000 UTC m=+66.206491677 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs") pod "network-metrics-daemon-tr4zt" (UID: "329df437-fb66-4775-a16e-94b06ef0cf80") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 01:26:25.642071 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.642033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:25.642558 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.642033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:25.642558 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.642172    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:25.642558 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.642284    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:25.642558 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:25.642037    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:25.642558 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:25.642384    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:27.642613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:27.642572    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:27.643081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:27.642622    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:27.643081 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:27.642728    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:27.643081 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:27.642766    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:27.643081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:27.642785    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:27.643081 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:27.642871    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:28.846465 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:28.846271    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="e47f38b52487122680b32a10cde4861b253fa52e1274f0c9908b1462447ef65a" exitCode=0
Jul 28 01:26:28.846881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:28.846358    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"e47f38b52487122680b32a10cde4861b253fa52e1274f0c9908b1462447ef65a"}
Jul 28 01:26:29.642566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:29.642530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:29.642566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:29.642571    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:29.642850 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:29.642530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:29.642850 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:29.642635    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:29.642850 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:29.642707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:29.642850 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:29.642788    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:29.851597 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:29.851562    2581 generic.go:356] "Generic (PLEG): container finished" podID="119d49ad-c20e-42b4-ba59-4fead814dd6e" containerID="898becee62afbd939c34f65da544cae7ec7ea9823440cdf1a3ac54cc2c5243e1" exitCode=0
Jul 28 01:26:29.851991 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:29.851613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerDied","Data":"898becee62afbd939c34f65da544cae7ec7ea9823440cdf1a3ac54cc2c5243e1"}
Jul 28 01:26:30.857286 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:30.857246    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" event={"ID":"119d49ad-c20e-42b4-ba59-4fead814dd6e","Type":"ContainerStarted","Data":"362a8c297e9e42e10c7c76633f784fb49f963d2512410d748a9205abb3e2d23b"}
Jul 28 01:26:31.643439 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:31.643402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:31.643697 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:31.643484    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:31.643697 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:31.643498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:31.643697 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:31.643595    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:31.643697 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:31.643628    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:31.643697 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:31.643690    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:33.039289 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.039253    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.233305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.233271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.421042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.420937    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.528907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.528877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:33.642591 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.642559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:33.642591 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.642591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:33.642845 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:33.642559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:33.642845 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:33.642686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:33.642845 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:33.642760    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:33.643008 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:33.642855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:34.131362 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.131239    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mvbd5" podStartSLOduration=13.276506748 podStartE2EDuration="43.131216033s" totalImagesPullingTime="29.854709285s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:30.886876895 +0000 UTC m=+39.792446292" watchObservedRunningTime="2026-07-28 01:26:34.131216033 +0000 UTC m=+43.036785434"
Jul 28 01:26:34.133055 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133018    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g79hr"]
Jul 28 01:26:34.133182 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tr4zt"]
Jul 28 01:26:34.133237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133196    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:34.133237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:34.133350 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:34.133326    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:34.133552 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hsjxk"]
Jul 28 01:26:34.133552 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:34.133555    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:34.133735 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.133605    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:34.133735 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:34.133692    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:34.365989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.365954    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:34.436329 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:34.436256    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:35.642528 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:35.642496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:35.643001 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:35.642493    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:35.643001 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:35.642505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:35.643001 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:35.642627    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:35.643001 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:35.642765    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:35.643001 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:35.642858    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:36.400703 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:36.400669    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:36.499894 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:36.499856    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:37.642598 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.642265    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:37.642598 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.642266    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:37.643042 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:37.642631    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g79hr" podUID="c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7"
Jul 28 01:26:37.643042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.642275    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:37.643042 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:37.642687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tr4zt" podUID="329df437-fb66-4775-a16e-94b06ef0cf80"
Jul 28 01:26:37.643042 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:37.642765    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hsjxk" podUID="9d8f5494-05fc-48a7-a8c2-cbfbf30134fb"
Jul 28 01:26:37.821367 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.821334    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-120.ec2.internal" event="NodeReady"
Jul 28 01:26:37.821505 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.821479    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 01:26:37.879932 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.879904    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-pld79"]
Jul 28 01:26:37.901661 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.901588    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kmhz4"]
Jul 28 01:26:37.901818 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.901715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:37.904415 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.904391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6wwkj\""
Jul 28 01:26:37.904415 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.904406    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 01:26:37.904633 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.904563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 01:26:37.904633 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.904631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.905028 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.905012    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.915344 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.915321    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pld79"]
Jul 28 01:26:37.915458 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.915358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmhz4"]
Jul 28 01:26:37.915458 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.915374    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q9h5v"]
Jul 28 01:26:37.915527 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.915510    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:37.919742 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.919721    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 01:26:37.920223 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.920203    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 01:26:37.920323 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.920272    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 01:26:37.920774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.920755    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zgp9f\""
Jul 28 01:26:37.931342 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.931314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q9h5v"]
Jul 28 01:26:37.931495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.931431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:37.934477 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.934456    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hjm4w\""
Jul 28 01:26:37.934626 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.934610    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 01:26:37.934991 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.934977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 01:26:37.988344 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.988304    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:37.988344 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.988344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w46fb\" (UniqueName: \"kubernetes.io/projected/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-api-access-w46fb\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:37.988572 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.988364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-crio-socket\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:37.988572 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.988394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-data-volume\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:37.988572 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:37.988416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.050968 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.050942    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:38.089048 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0806ff13-11a2-41c7-8c0b-84f6788b459a-metrics-tls\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.089188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2dhp\" (UniqueName: \"kubernetes.io/projected/0806ff13-11a2-41c7-8c0b-84f6788b459a-kube-api-access-k2dhp\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.089188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089099    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-data-volume\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089261    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0806ff13-11a2-41c7-8c0b-84f6788b459a-config-volume\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.089397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/962dc654-599f-420d-979a-95ce484e7b2a-cert\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.089397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089351    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0806ff13-11a2-41c7-8c0b-84f6788b459a-tmp-dir\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.089490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089413    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-data-volume\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w46fb\" (UniqueName: \"kubernetes.io/projected/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-api-access-w46fb\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h88hm\" (UniqueName: \"kubernetes.io/projected/962dc654-599f-420d-979a-95ce484e7b2a-kube-api-access-h88hm\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.089692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089510    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-crio-socket\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.089750 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-crio-socket\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.090000 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.089980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.093316 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.093298    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.106389 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.106360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w46fb\" (UniqueName: \"kubernetes.io/projected/34b5cb41-e7b0-4e5d-9962-7ee44bba7914-kube-api-access-w46fb\") pod \"insights-runtime-extractor-pld79\" (UID: \"34b5cb41-e7b0-4e5d-9962-7ee44bba7914\") " pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.151510 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.151477    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:38.190712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190625    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h88hm\" (UniqueName: \"kubernetes.io/projected/962dc654-599f-420d-979a-95ce484e7b2a-kube-api-access-h88hm\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.190712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0806ff13-11a2-41c7-8c0b-84f6788b459a-metrics-tls\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.190712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190686    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2dhp\" (UniqueName: \"kubernetes.io/projected/0806ff13-11a2-41c7-8c0b-84f6788b459a-kube-api-access-k2dhp\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.190995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0806ff13-11a2-41c7-8c0b-84f6788b459a-config-volume\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.190995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/962dc654-599f-420d-979a-95ce484e7b2a-cert\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.190995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.190818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0806ff13-11a2-41c7-8c0b-84f6788b459a-tmp-dir\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.191237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.191217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0806ff13-11a2-41c7-8c0b-84f6788b459a-tmp-dir\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.191359 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.191333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0806ff13-11a2-41c7-8c0b-84f6788b459a-config-volume\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.193172 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.193155    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/962dc654-599f-420d-979a-95ce484e7b2a-cert\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.193227 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.193167    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0806ff13-11a2-41c7-8c0b-84f6788b459a-metrics-tls\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.198993 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.198970    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2dhp\" (UniqueName: \"kubernetes.io/projected/0806ff13-11a2-41c7-8c0b-84f6788b459a-kube-api-access-k2dhp\") pod \"dns-default-q9h5v\" (UID: \"0806ff13-11a2-41c7-8c0b-84f6788b459a\") " pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.199242 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.199218    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h88hm\" (UniqueName: \"kubernetes.io/projected/962dc654-599f-420d-979a-95ce484e7b2a-kube-api-access-h88hm\") pod \"ingress-canary-kmhz4\" (UID: \"962dc654-599f-420d-979a-95ce484e7b2a\") " pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.211548 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.211518    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-pld79"
Jul 28 01:26:38.224339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.224315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kmhz4"
Jul 28 01:26:38.239624 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.239589    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:38.376912 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.376878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-pld79"]
Jul 28 01:26:38.380664 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:38.380633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b5cb41_e7b0_4e5d_9962_7ee44bba7914.slice/crio-2115f193d679c19b4706368fd16b0cb7d4859f89746aba283ab832f57fac0ef5 WatchSource:0}: Error finding container 2115f193d679c19b4706368fd16b0cb7d4859f89746aba283ab832f57fac0ef5: Status 404 returned error can't find the container with id 2115f193d679c19b4706368fd16b0cb7d4859f89746aba283ab832f57fac0ef5
Jul 28 01:26:38.383571 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.383543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kmhz4"]
Jul 28 01:26:38.387540 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:38.387510    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod962dc654_599f_420d_979a_95ce484e7b2a.slice/crio-b289fc8ddb27308449c8407deed56cc199c134c670814788217ec9b490d0cfb4 WatchSource:0}: Error finding container b289fc8ddb27308449c8407deed56cc199c134c670814788217ec9b490d0cfb4: Status 404 returned error can't find the container with id b289fc8ddb27308449c8407deed56cc199c134c670814788217ec9b490d0cfb4
Jul 28 01:26:38.397822 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.397783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q9h5v"]
Jul 28 01:26:38.401473 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:38.401441    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0806ff13_11a2_41c7_8c0b_84f6788b459a.slice/crio-eb07f74c3150a09b0397f5fd50cae4bff9ec1f675b639c0a737b0aaeac403ccd WatchSource:0}: Error finding container eb07f74c3150a09b0397f5fd50cae4bff9ec1f675b639c0a737b0aaeac403ccd: Status 404 returned error can't find the container with id eb07f74c3150a09b0397f5fd50cae4bff9ec1f675b639c0a737b0aaeac403ccd
Jul 28 01:26:38.883685 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.883647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9h5v" event={"ID":"0806ff13-11a2-41c7-8c0b-84f6788b459a","Type":"ContainerStarted","Data":"eb07f74c3150a09b0397f5fd50cae4bff9ec1f675b639c0a737b0aaeac403ccd"}
Jul 28 01:26:38.885366 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.885303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmhz4" event={"ID":"962dc654-599f-420d-979a-95ce484e7b2a","Type":"ContainerStarted","Data":"b289fc8ddb27308449c8407deed56cc199c134c670814788217ec9b490d0cfb4"}
Jul 28 01:26:38.886568 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:38.886538    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pld79" event={"ID":"34b5cb41-e7b0-4e5d-9962-7ee44bba7914","Type":"ContainerStarted","Data":"2115f193d679c19b4706368fd16b0cb7d4859f89746aba283ab832f57fac0ef5"}
Jul 28 01:26:39.642814 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.642766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:39.643163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.643140    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:39.643293 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.643269    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:39.646482 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.646297    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:39.648595 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.648358    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lwsn\""
Jul 28 01:26:39.648595 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.648404    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8db89\""
Jul 28 01:26:39.648595 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.648463    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:39.648838 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.648650    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:39.648838 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:39.648726    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:40.218147 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.218112    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.314528 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.314258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.403149 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.403083    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.523112 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.523076    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.529745 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.529714    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-hw9f2"]
Jul 28 01:26:40.551834 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.551765    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-hw9f2"]
Jul 28 01:26:40.552005 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.551928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:26:40.555435 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.555411    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 01:26:40.555897 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.555872    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fr4zh\""
Jul 28 01:26:40.555897 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.555877    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 01:26:40.619984 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.619750    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.711670 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.711634    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9xj\" (UniqueName: \"kubernetes.io/projected/886e9d54-5d8a-4d0d-b1af-bd2963f51ac4-kube-api-access-2r9xj\") pod \"downloads-7c97dc4555-hw9f2\" (UID: \"886e9d54-5d8a-4d0d-b1af-bd2963f51ac4\") " pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:26:40.719055 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.719001    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.812955 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.812869    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9xj\" (UniqueName: \"kubernetes.io/projected/886e9d54-5d8a-4d0d-b1af-bd2963f51ac4-kube-api-access-2r9xj\") pod \"downloads-7c97dc4555-hw9f2\" (UID: \"886e9d54-5d8a-4d0d-b1af-bd2963f51ac4\") " pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:26:40.820759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.820702    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:40.829412 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.829341    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9xj\" (UniqueName: \"kubernetes.io/projected/886e9d54-5d8a-4d0d-b1af-bd2963f51ac4-kube-api-access-2r9xj\") pod \"downloads-7c97dc4555-hw9f2\" (UID: \"886e9d54-5d8a-4d0d-b1af-bd2963f51ac4\") " pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:26:40.864565 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:40.864530    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:26:41.151016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.150931    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.218320 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.218280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.294596 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.294562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.572189 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.572157    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.674221 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.674187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.773110 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.773078    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.838604 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.838529    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:41.971614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:41.971581    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:42.371305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:42.371267    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:42.770187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:42.770147    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.380311 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:44.380235    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:44.437818 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:44.437767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.200759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.200523    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-hw9f2"]
Jul 28 01:26:45.214914 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:45.214880    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886e9d54_5d8a_4d0d_b1af_bd2963f51ac4.slice/crio-ba0102990755443e88a18db145a4cf78a473da922b3e246f18cb07ce1f053187 WatchSource:0}: Error finding container ba0102990755443e88a18db145a4cf78a473da922b3e246f18cb07ce1f053187: Status 404 returned error can't find the container with id ba0102990755443e88a18db145a4cf78a473da922b3e246f18cb07ce1f053187
Jul 28 01:26:45.441325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.441291    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.495582 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.495550    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"]
Jul 28 01:26:45.510493 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.510456    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.511251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.511225    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"]
Jul 28 01:26:45.513563 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.513400    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 01:26:45.513563 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.513532    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 01:26:45.516153 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.516136    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 01:26:45.516319 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.516299    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-jm8mp\""
Jul 28 01:26:45.516558 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.516508    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 01:26:45.516652 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.516583    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 01:26:45.537640 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.537562    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.553788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.553760    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.553788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.553791    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05176fac-afd9-4b0a-baca-04a1fdf0a260-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.554044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.553889    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.554044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.553954    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjsgb\" (UniqueName: \"kubernetes.io/projected/05176fac-afd9-4b0a-baca-04a1fdf0a260-kube-api-access-gjsgb\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.654865 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.654825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.654865 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.654862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjsgb\" (UniqueName: \"kubernetes.io/projected/05176fac-afd9-4b0a-baca-04a1fdf0a260-kube-api-access-gjsgb\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.655099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.654931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.655099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.654953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05176fac-afd9-4b0a-baca-04a1fdf0a260-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.655099 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:45.655056    2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 01:26:45.655234 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:45.655123    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls podName:05176fac-afd9-4b0a-baca-04a1fdf0a260 nodeName:}" failed. No retries permitted until 2026-07-28 01:26:46.155104834 +0000 UTC m=+55.060674218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-ptrd2" (UID: "05176fac-afd9-4b0a-baca-04a1fdf0a260") : secret "prometheus-operator-tls" not found
Jul 28 01:26:45.655650 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.655627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/05176fac-afd9-4b0a-baca-04a1fdf0a260-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.658791 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.658760    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.663755 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.663732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjsgb\" (UniqueName: \"kubernetes.io/projected/05176fac-afd9-4b0a-baca-04a1fdf0a260-kube-api-access-gjsgb\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:45.913604 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.913519    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-hw9f2" event={"ID":"886e9d54-5d8a-4d0d-b1af-bd2963f51ac4","Type":"ContainerStarted","Data":"ba0102990755443e88a18db145a4cf78a473da922b3e246f18cb07ce1f053187"}
Jul 28 01:26:45.915854 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.915819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9h5v" event={"ID":"0806ff13-11a2-41c7-8c0b-84f6788b459a","Type":"ContainerStarted","Data":"c4a6914128ceb5e27d1323fd416e74abd56b73b917dba349d0f517db23bb0e25"}
Jul 28 01:26:45.916025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.915880    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q9h5v" event={"ID":"0806ff13-11a2-41c7-8c0b-84f6788b459a","Type":"ContainerStarted","Data":"c57b9ec40559e659d83cd996f42eaf35f3420c1590e11b376b5207f702c31a79"}
Jul 28 01:26:45.916025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.915979    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:45.917555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.917474    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:45.917555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.917526    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kmhz4" event={"ID":"962dc654-599f-420d-979a-95ce484e7b2a","Type":"ContainerStarted","Data":"e7bfbaf873d5f5b6384d8be6d66e380e46a4deb94c5107863ecf4337bcbb8348"}
Jul 28 01:26:45.919279 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.919252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pld79" event={"ID":"34b5cb41-e7b0-4e5d-9962-7ee44bba7914","Type":"ContainerStarted","Data":"d70312d947de5d116453b39227b974a3cd3247280eb9476be015dcab56fdc047"}
Jul 28 01:26:45.933824 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.933743    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q9h5v" podStartSLOduration=2.303904846 podStartE2EDuration="8.933720541s" totalImagesPullingTime="6.629815695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:45.932685568 +0000 UTC m=+54.838254994" watchObservedRunningTime="2026-07-28 01:26:45.933720541 +0000 UTC m=+54.839289935"
Jul 28 01:26:45.949936 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:45.949871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kmhz4" podStartSLOduration=2.733952178 podStartE2EDuration="8.949846132s" totalImagesPullingTime="6.215893954s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:45.949126233 +0000 UTC m=+54.854695634" watchObservedRunningTime="2026-07-28 01:26:45.949846132 +0000 UTC m=+54.855415530"
Jul 28 01:26:46.159895 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:46.159255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:46.165775 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:46.165649    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/05176fac-afd9-4b0a-baca-04a1fdf0a260-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-ptrd2\" (UID: \"05176fac-afd9-4b0a-baca-04a1fdf0a260\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:46.421005 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:46.420600    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"
Jul 28 01:26:46.603857 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:46.603821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-ptrd2"]
Jul 28 01:26:46.609305 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:46.609271    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05176fac_afd9_4b0a_baca_04a1fdf0a260.slice/crio-8f0d7ac751372cba297cda65c4bcb531abb3dc8eadcbccaf529cff0f1f6383ea WatchSource:0}: Error finding container 8f0d7ac751372cba297cda65c4bcb531abb3dc8eadcbccaf529cff0f1f6383ea: Status 404 returned error can't find the container with id 8f0d7ac751372cba297cda65c4bcb531abb3dc8eadcbccaf529cff0f1f6383ea
Jul 28 01:26:46.925074 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:46.924761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2" event={"ID":"05176fac-afd9-4b0a-baca-04a1fdf0a260","Type":"ContainerStarted","Data":"8f0d7ac751372cba297cda65c4bcb531abb3dc8eadcbccaf529cff0f1f6383ea"}
Jul 28 01:26:47.748143 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:47.748096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:47.855269 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:47.855231    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.765689 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:49.765640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:49.867479 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:49.867441    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.504186 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.504142    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:50.824257 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.824157    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:26:50.826717 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.826689    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.834354 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.834300    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 01:26:50.834675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.834472    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 01:26:50.834817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.834767    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 01:26:50.835027 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.835005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 01:26:50.835105 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.835059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7m25v\""
Jul 28 01:26:50.835692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.835665    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 01:26:50.846929 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.846901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:26:50.899731 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.899731 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899732    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.900012 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899753    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.900012 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899773    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.900012 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.900012 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.899933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:50.938520 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.938477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pld79" event={"ID":"34b5cb41-e7b0-4e5d-9962-7ee44bba7914","Type":"ContainerStarted","Data":"4f0481077e9cba4ba773f72a513eaf6af0d7d24d27eb1f19a4203f124c59eb6a"}
Jul 28 01:26:50.940560 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.940508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2" event={"ID":"05176fac-afd9-4b0a-baca-04a1fdf0a260","Type":"ContainerStarted","Data":"980f2fe16b7d45b4b6d8e1b1a0dd3aaff48125d43bcda3f3c3520d9cae1b90b7"}
Jul 28 01:26:50.940717 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.940566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2" event={"ID":"05176fac-afd9-4b0a-baca-04a1fdf0a260","Type":"ContainerStarted","Data":"b5144514319ae653d1a3fbe110dcff582d02fd5f4509e0f2aa4b42e3aaeac311"}
Jul 28 01:26:50.973272 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:50.973212    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-ptrd2" podStartSLOduration=2.566132249 podStartE2EDuration="5.973191266s" totalImagesPullingTime="3.407059017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:50.972506238 +0000 UTC m=+59.878075637" watchObservedRunningTime="2026-07-28 01:26:50.973191266 +0000 UTC m=+59.878760645"
Jul 28 01:26:51.001015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.000979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.001201 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.001201 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001155    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.001314 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001241    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.001314 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001289    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.001410 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.002004 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001784    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.002004 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.001907    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.002173 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.002105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.005583 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.005555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.006173 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.006150    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.028176 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.028133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs\") pod \"console-74665dbc9f-4mrgl\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") " pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.065626 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.065592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.144900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.144817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:26:51.163297 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.162590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.337841 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.337788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:26:51.342063 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:51.342029    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ddfd47_339a_46d8_bdc9_8d6a8a26434b.slice/crio-da30d3e60b4f9b67b4f1323fd89fb072b89fa090e62db8795d7257e9fabe8438 WatchSource:0}: Error finding container da30d3e60b4f9b67b4f1323fd89fb072b89fa090e62db8795d7257e9fabe8438: Status 404 returned error can't find the container with id da30d3e60b4f9b67b4f1323fd89fb072b89fa090e62db8795d7257e9fabe8438
Jul 28 01:26:51.421652 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.421548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.540671 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.540636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:51.836654 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.836622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-zkwqs"
Jul 28 01:26:51.947217 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:51.947170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74665dbc9f-4mrgl" event={"ID":"74ddfd47-339a-46d8-bdc9-8d6a8a26434b","Type":"ContainerStarted","Data":"da30d3e60b4f9b67b4f1323fd89fb072b89fa090e62db8795d7257e9fabe8438"}
Jul 28 01:26:52.091065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:52.090968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.238782 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:52.238659    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.673825 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:52.673727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:52.988310 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:52.988186    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.100507 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.100466    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"]
Jul 28 01:26:53.176890 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.176852    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"]
Jul 28 01:26:53.178375 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.177170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.180141 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.180080    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-7dg5j\""
Jul 28 01:26:53.181441 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.181037    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 28 01:26:53.181441 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.181267    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 28 01:26:53.222689 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.222653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.222888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.222713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.222888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.222821    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4e918-20c2-4293-acb8-d32ada84f2eb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.222888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.222863    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hx2l\" (UniqueName: \"kubernetes.io/projected/3ed4e918-20c2-4293-acb8-d32ada84f2eb-kube-api-access-5hx2l\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.228080 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.227195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"]
Jul 28 01:26:53.228080 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.227252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"]
Jul 28 01:26:53.228080 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.227287    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2qq55"]
Jul 28 01:26:53.228887 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.228498    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.256159 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.256126    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wls6v\""
Jul 28 01:26:53.256746 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.256724    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 01:26:53.256876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.256759    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 01:26:53.257136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.257117    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 01:26:53.263406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.263340    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.270048 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.270019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.274325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.273096    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 01:26:53.274325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.273486    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mgxfm\""
Jul 28 01:26:53.274325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.274051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 01:26:53.274325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.274268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 01:26:53.323495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hx2l\" (UniqueName: \"kubernetes.io/projected/3ed4e918-20c2-4293-acb8-d32ada84f2eb-kube-api-access-5hx2l\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323510    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk77m\" (UniqueName: \"kubernetes.io/projected/f1c85108-418d-4c11-8762-8231d4f84826-kube-api-access-vk77m\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323547    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-textfile\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323625    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323641    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0dd6710-e852-46bb-991c-cac90d69c12d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.323673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-sys\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323685    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-root\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323722    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-tls\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4e918-20c2-4293-acb8-d32ada84f2eb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-metrics-client-ca\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323789    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323833    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323869    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6vl\" (UniqueName: \"kubernetes.io/projected/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-api-access-sw6vl\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-wtmp\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.324081 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.323904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.324738 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.324703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.324950 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.324924    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ed4e918-20c2-4293-acb8-d32ada84f2eb-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.325079 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:53.325065    2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Jul 28 01:26:53.325139 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:53.325118    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls podName:3ed4e918-20c2-4293-acb8-d32ada84f2eb nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.825103655 +0000 UTC m=+62.730673031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-jxxst" (UID: "3ed4e918-20c2-4293-acb8-d32ada84f2eb") : secret "openshift-state-metrics-tls" not found
Jul 28 01:26:53.328786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.328724    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.357910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.357281    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hx2l\" (UniqueName: \"kubernetes.io/projected/3ed4e918-20c2-4293-acb8-d32ada84f2eb-kube-api-access-5hx2l\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.425924 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.425882    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6vl\" (UniqueName: \"kubernetes.io/projected/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-api-access-sw6vl\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.426115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.425929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-wtmp\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426015    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk77m\" (UniqueName: \"kubernetes.io/projected/f1c85108-418d-4c11-8762-8231d4f84826-kube-api-access-vk77m\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-textfile\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426122    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0dd6710-e852-46bb-991c-cac90d69c12d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-sys\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426221    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-root\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-tls\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-metrics-client-ca\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.426372 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.426358    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:53.426945    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:26:53.427032    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls podName:c0dd6710-e852-46bb-991c-cac90d69c12d nodeName:}" failed. No retries permitted until 2026-07-28 01:26:53.927012537 +0000 UTC m=+62.832581919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls") pod "kube-state-metrics-56d9f944b4-lmxt4" (UID: "c0dd6710-e852-46bb-991c-cac90d69c12d") : secret "kube-state-metrics-tls" not found
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.427221    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.427505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c0dd6710-e852-46bb-991c-cac90d69c12d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.427562    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-sys\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.427612    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-wtmp\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.428146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c0dd6710-e852-46bb-991c-cac90d69c12d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.428211    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f1c85108-418d-4c11-8762-8231d4f84826-root\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.428770    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-accelerators-collector-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.429041    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-textfile\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.429881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.429511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1c85108-418d-4c11-8762-8231d4f84826-metrics-client-ca\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.431594 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.431193    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.431594 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.431550    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f1c85108-418d-4c11-8762-8231d4f84826-node-exporter-tls\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.433285 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.433261    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.444335 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.444271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk77m\" (UniqueName: \"kubernetes.io/projected/f1c85108-418d-4c11-8762-8231d4f84826-kube-api-access-vk77m\") pod \"node-exporter-2qq55\" (UID: \"f1c85108-418d-4c11-8762-8231d4f84826\") " pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.448248 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.448187    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6vl\" (UniqueName: \"kubernetes.io/projected/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-api-access-sw6vl\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.459483 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.459389    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.586036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.585948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2qq55"
Jul 28 01:26:53.606958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.606925    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.778500 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.778434    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.830092 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.830057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.833009 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.832977    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3ed4e918-20c2-4293-acb8-d32ada84f2eb-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-jxxst\" (UID: \"3ed4e918-20c2-4293-acb8-d32ada84f2eb\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:53.919527 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.919448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:53.931712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.931676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:53.934675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:53.934645    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c0dd6710-e852-46bb-991c-cac90d69c12d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-lmxt4\" (UID: \"c0dd6710-e852-46bb-991c-cac90d69c12d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:54.074684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:54.074607    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.091535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:54.091501    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"
Jul 28 01:26:54.142634 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:54.142596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"
Jul 28 01:26:54.266488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:54.266448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:54.540202 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:54.540097    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:55.868298 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:55.868259    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c85108_418d_4c11_8762_8231d4f84826.slice/crio-7d6a012fd08025ef46c2387631dc1809b38ad2ae3f2b5894783239875f5c908f WatchSource:0}: Error finding container 7d6a012fd08025ef46c2387631dc1809b38ad2ae3f2b5894783239875f5c908f: Status 404 returned error can't find the container with id 7d6a012fd08025ef46c2387631dc1809b38ad2ae3f2b5894783239875f5c908f
Jul 28 01:26:55.928207 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:55.927600    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q9h5v"
Jul 28 01:26:55.971831 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:55.971273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74665dbc9f-4mrgl" event={"ID":"74ddfd47-339a-46d8-bdc9-8d6a8a26434b","Type":"ContainerStarted","Data":"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"}
Jul 28 01:26:55.975353 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:55.975268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-pld79" event={"ID":"34b5cb41-e7b0-4e5d-9962-7ee44bba7914","Type":"ContainerStarted","Data":"4f876b5f80a7feadc0fbdee1fce6ca40064108960d2186494d43fcf0fd7733ec"}
Jul 28 01:26:55.978529 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:55.978438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qq55" event={"ID":"f1c85108-418d-4c11-8762-8231d4f84826","Type":"ContainerStarted","Data":"7d6a012fd08025ef46c2387631dc1809b38ad2ae3f2b5894783239875f5c908f"}
Jul 28 01:26:56.037836 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.036268    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-pld79" podStartSLOduration=1.7315614510000001 podStartE2EDuration="19.036250627s" totalImagesPullingTime="17.304689176s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:56.036095967 +0000 UTC m=+64.941665364" watchObservedRunningTime="2026-07-28 01:26:56.036250627 +0000 UTC m=+64.941820015"
Jul 28 01:26:56.037836 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.036393    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74665dbc9f-4mrgl" podStartSLOduration=1.50734008 podStartE2EDuration="6.036388814s" totalImagesPullingTime="4.529048734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:26:56.003967265 +0000 UTC m=+64.909536673" watchObservedRunningTime="2026-07-28 01:26:56.036388814 +0000 UTC m=+64.941958214"
Jul 28 01:26:56.039874 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.039621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst"]
Jul 28 01:26:56.042558 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:56.042523    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed4e918_20c2_4293_acb8_d32ada84f2eb.slice/crio-bf2d9178298b564d27b1fa4f3f2ee672cbad16d61a5236b05221b0bf9181c326 WatchSource:0}: Error finding container bf2d9178298b564d27b1fa4f3f2ee672cbad16d61a5236b05221b0bf9181c326: Status 404 returned error can't find the container with id bf2d9178298b564d27b1fa4f3f2ee672cbad16d61a5236b05221b0bf9181c326
Jul 28 01:26:56.065352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.065263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4"]
Jul 28 01:26:56.078710 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:26:56.078668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0dd6710_e852_46bb_991c_cac90d69c12d.slice/crio-e244c44de983720e5e141b1b9ded137eeba8a8eede57eb84b35b2457d2c972c2 WatchSource:0}: Error finding container e244c44de983720e5e141b1b9ded137eeba8a8eede57eb84b35b2457d2c972c2: Status 404 returned error can't find the container with id e244c44de983720e5e141b1b9ded137eeba8a8eede57eb84b35b2457d2c972c2
Jul 28 01:26:56.087971 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.087903    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.193323 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.193279    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.303238 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.303184    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.455124 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.455042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:56.983842 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.983806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst" event={"ID":"3ed4e918-20c2-4293-acb8-d32ada84f2eb","Type":"ContainerStarted","Data":"8c757627b73c349f147dccf9bc7dae040e02ae066295233807092ac104b900f6"}
Jul 28 01:26:56.984323 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.983902    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst" event={"ID":"3ed4e918-20c2-4293-acb8-d32ada84f2eb","Type":"ContainerStarted","Data":"b0496aa5cecd76f11a6e63a973a1c0a4e845b89dcdadb669a5ce594455df7658"}
Jul 28 01:26:56.984323 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.983935    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst" event={"ID":"3ed4e918-20c2-4293-acb8-d32ada84f2eb","Type":"ContainerStarted","Data":"bf2d9178298b564d27b1fa4f3f2ee672cbad16d61a5236b05221b0bf9181c326"}
Jul 28 01:26:56.985151 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:56.985104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4" event={"ID":"c0dd6710-e852-46bb-991c-cac90d69c12d","Type":"ContainerStarted","Data":"e244c44de983720e5e141b1b9ded137eeba8a8eede57eb84b35b2457d2c972c2"}
Jul 28 01:26:57.197740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.197701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.265012 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.264972    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:57.265219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.265036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:57.267777 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.267754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 01:26:57.268350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.268327    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 01:26:57.274165 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.274137    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.278379 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.278354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9d8f5494-05fc-48a7-a8c2-cbfbf30134fb-original-pull-secret\") pod \"global-pull-secret-syncer-hsjxk\" (UID: \"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb\") " pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:57.280967 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.280948    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 01:26:57.288808 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.288767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfsfb\" (UniqueName: \"kubernetes.io/projected/c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7-kube-api-access-xfsfb\") pod \"network-check-target-g79hr\" (UID: \"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7\") " pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:57.359460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.359427    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hsjxk"
Jul 28 01:26:57.366014 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.365988    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:57.368960 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.368935    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 01:26:57.379621 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.379594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/329df437-fb66-4775-a16e-94b06ef0cf80-metrics-certs\") pod \"network-metrics-daemon-tr4zt\" (UID: \"329df437-fb66-4775-a16e-94b06ef0cf80\") " pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:57.380518 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.380491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-4lwsn\""
Jul 28 01:26:57.387740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.387715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:26:57.668346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.668252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.669550 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.669524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8db89\""
Jul 28 01:26:57.678112 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.678080    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tr4zt"
Jul 28 01:26:57.686043 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.686015    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"]
Jul 28 01:26:57.688282 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.688257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.691827 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.691789    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-4s4vh\""
Jul 28 01:26:57.691995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.691816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 28 01:26:57.704949 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.704922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"]
Jul 28 01:26:57.768712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.768661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a780f93f-13ea-4bd5-b370-64c18d4e5c60-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.769072 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.768910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gg57\" (UniqueName: \"kubernetes.io/projected/a780f93f-13ea-4bd5-b370-64c18d4e5c60-kube-api-access-8gg57\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.769788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.769741    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:57.869711 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.869667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a780f93f-13ea-4bd5-b370-64c18d4e5c60-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.869897 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.869733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gg57\" (UniqueName: \"kubernetes.io/projected/a780f93f-13ea-4bd5-b370-64c18d4e5c60-kube-api-access-8gg57\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.872477 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.872444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/a780f93f-13ea-4bd5-b370-64c18d4e5c60-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.879123 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.879094    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gg57\" (UniqueName: \"kubernetes.io/projected/a780f93f-13ea-4bd5-b370-64c18d4e5c60-kube-api-access-8gg57\") pod \"monitoring-plugin-68b7fc9574-vbj2j\" (UID: \"a780f93f-13ea-4bd5-b370-64c18d4e5c60\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:57.915627 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:57.915575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.000881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.000779    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:26:58.006270 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.006245    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.409213 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.409183    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7fc868b465-cpngh"]
Jul 28 01:26:58.411728 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.411702    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.416287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.416258    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 01:26:58.416442 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.416307    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 01:26:58.416625 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.416603    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-khtjr\""
Jul 28 01:26:58.416678 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.416661    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 01:26:58.417592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.417570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 01:26:58.418793 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.418775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 01:26:58.423021 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.422996    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 01:26:58.441561 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.441530    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc868b465-cpngh"]
Jul 28 01:26:58.451083 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.450997    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.476152 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-federate-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476348 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476348 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476204    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476348 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476231    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-metrics-client-ca\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476348 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476287    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476348 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476579 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476370    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j59d2\" (UniqueName: \"kubernetes.io/projected/2ac8110e-26ae-412f-8188-bd530bf3f0d4-kube-api-access-j59d2\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.476579 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.476431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.514560 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.514525    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:58.577079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-metrics-client-ca\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577157    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577195    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j59d2\" (UniqueName: \"kubernetes.io/projected/2ac8110e-26ae-412f-8188-bd530bf3f0d4-kube-api-access-j59d2\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.577336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.577271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-federate-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.578618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.578533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-metrics-client-ca\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.578618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.578558    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-serving-certs-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.578783 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.578650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.580212 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.580186    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.580394 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.580373    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-federate-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.580954 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.580931    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-telemeter-client-tls\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.581091 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.581076    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/2ac8110e-26ae-412f-8188-bd530bf3f0d4-secret-telemeter-client\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.596302 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.596270    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j59d2\" (UniqueName: \"kubernetes.io/projected/2ac8110e-26ae-412f-8188-bd530bf3f0d4-kube-api-access-j59d2\") pod \"telemeter-client-7fc868b465-cpngh\" (UID: \"2ac8110e-26ae-412f-8188-bd530bf3f0d4\") " pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:58.724538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:58.724446    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh"
Jul 28 01:26:59.355854 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.354486    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:26:59.360071 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.360040    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:26:59.382058 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.382015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.386740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.385533    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 01:26:59.386740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.386425    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 01:26:59.386740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.386444    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 01:26:59.387390 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.386820    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 01:26:59.387985 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.387963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 01:26:59.388266 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.388247    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 01:26:59.388555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.388536    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 01:26:59.389078 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.388839    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 01:26:59.390133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.389881    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-clfv5bml756ih\""
Jul 28 01:26:59.390133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.389929    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 01:26:59.390133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390086    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 01:26:59.390908 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390885    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7fd5v\""
Jul 28 01:26:59.390997 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390350    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 01:26:59.391062 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390424    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:26:59.391119 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390291    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 01:26:59.391119 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 01:26:59.391243 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.390587    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 01:26:59.392020 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.391637    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 01:26:59.484874 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.484835    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.484894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.484921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.484949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.484980    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485034    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485064    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485152    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485213    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485258    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485425    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485459    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-995jm\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.485730 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.485574    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586771    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586827    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586855    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586934    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586960    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.586986    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587036    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587123    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.587989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.588929 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.588929 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587287    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.588929 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.587326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-995jm\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.589832 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.589474    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.589832 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.589715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.589832 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.589821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.591900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.591753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.592901 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.592867    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.593074 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.593050    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.593638 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.593265    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.593638 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.593559    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.594340 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.594291    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.594340 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.594321    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.595584 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.595560    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.596566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.596521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.597360 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.597335    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.597918 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.597896    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-995jm\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.598698 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.598671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.599236 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.599171    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.599399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.599378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.601304 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.600980    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.601304 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.601255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.601746 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.601719    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:26:59.697884 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:26:59.697781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:01.145336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:01.145297    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:01.145786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:01.145403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:01.151065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:01.151033    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:02.011180 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:02.011139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:03.153406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.153370    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.288927 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.288890    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.401219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.401182    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.468370 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.468290    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:03.799184 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.799136    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tr4zt"]
Jul 28 01:27:03.867768 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.867666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:27:03.881720 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:27:03.881683    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329df437_fb66_4775_a16e_94b06ef0cf80.slice/crio-020eaa78a64ea45251362fb10691be01fe980873a0c882af3d7b0f5f180e50b5 WatchSource:0}: Error finding container 020eaa78a64ea45251362fb10691be01fe980873a0c882af3d7b0f5f180e50b5: Status 404 returned error can't find the container with id 020eaa78a64ea45251362fb10691be01fe980873a0c882af3d7b0f5f180e50b5
Jul 28 01:27:03.882079 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:27:03.882051    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a8ffc98_be42_4cc0_b55d_e483adbd9b55.slice/crio-a9ab821047765f87d359e6fffd29c8238faa58bceda3c394fa02a99ebe869392 WatchSource:0}: Error finding container a9ab821047765f87d359e6fffd29c8238faa58bceda3c394fa02a99ebe869392: Status 404 returned error can't find the container with id a9ab821047765f87d359e6fffd29c8238faa58bceda3c394fa02a99ebe869392
Jul 28 01:27:03.992850 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:03.992763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:04.020727 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.020628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"a9ab821047765f87d359e6fffd29c8238faa58bceda3c394fa02a99ebe869392"}
Jul 28 01:27:04.026665 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.025296    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr4zt" event={"ID":"329df437-fb66-4775-a16e-94b06ef0cf80","Type":"ContainerStarted","Data":"020eaa78a64ea45251362fb10691be01fe980873a0c882af3d7b0f5f180e50b5"}
Jul 28 01:27:04.028605 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.028497    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g79hr"]
Jul 28 01:27:04.029850 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.029684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hsjxk"]
Jul 28 01:27:04.030446 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:27:04.030369    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bd1caa_d681_4e7c_9361_e5fba9a8aaa7.slice/crio-7ef7ed11090f8f3910874e9bdfd6ce2dbe347e2ec2cd646bb4bc783b46611b8b WatchSource:0}: Error finding container 7ef7ed11090f8f3910874e9bdfd6ce2dbe347e2ec2cd646bb4bc783b46611b8b: Status 404 returned error can't find the container with id 7ef7ed11090f8f3910874e9bdfd6ce2dbe347e2ec2cd646bb4bc783b46611b8b
Jul 28 01:27:04.037768 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.034431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7fc868b465-cpngh"]
Jul 28 01:27:04.045149 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.042673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"]
Jul 28 01:27:04.095420 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:04.095341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:05.033123 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.032874    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-hw9f2" event={"ID":"886e9d54-5d8a-4d0d-b1af-bd2963f51ac4","Type":"ContainerStarted","Data":"6852bb541ed9a6ab92dd8304afeda80feedce23cf34bf9c9d7007b11ae908c20"}
Jul 28 01:27:05.034359 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.034290    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:27:05.037170 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.037143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh" event={"ID":"2ac8110e-26ae-412f-8188-bd530bf3f0d4","Type":"ContainerStarted","Data":"65fd8e99aa111be337aed6b34ecbcc15f169cb2de4fa78879d582131f72bd62d"}
Jul 28 01:27:05.038982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.038945    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j" event={"ID":"a780f93f-13ea-4bd5-b370-64c18d4e5c60","Type":"ContainerStarted","Data":"73c46b79be1558a4ee5a066a07698546d58ce50dda1eec07c23ac7b66eb7a8c8"}
Jul 28 01:27:05.041192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.041149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g79hr" event={"ID":"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7","Type":"ContainerStarted","Data":"7ef7ed11090f8f3910874e9bdfd6ce2dbe347e2ec2cd646bb4bc783b46611b8b"}
Jul 28 01:27:05.044978 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.044930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hsjxk" event={"ID":"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb","Type":"ContainerStarted","Data":"970fe0f961e3247df91909f20fe696563edbd8917ba1c5f707275cc56155fb2f"}
Jul 28 01:27:05.050414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.050322    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-hw9f2"
Jul 28 01:27:05.053321 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.053035    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-hw9f2" podStartSLOduration=6.3407577459999995 podStartE2EDuration="25.05301722s" totalImagesPullingTime="18.712259474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:05.051725535 +0000 UTC m=+73.957294934" watchObservedRunningTime="2026-07-28 01:27:05.05301722 +0000 UTC m=+73.958586602"
Jul 28 01:27:05.581645 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.581613    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:05.646289 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:05.646111    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:06.386907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:06.386819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:06.483547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:06.483424    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:07.511415 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:07.511378    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:07.597309 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:07.597278    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:08.989301 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:08.989225    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:09.090783 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:09.090634    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:10.749854 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:10.749390    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:27:12.557192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:12.557154    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:12.631756 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:12.631636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.809398 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:13.809355    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:13.881361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:13.881318    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:14.391171 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:14.391131    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:14.492398 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:14.492366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:15.098702 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.098557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr4zt" event={"ID":"329df437-fb66-4775-a16e-94b06ef0cf80","Type":"ContainerStarted","Data":"bf7e4b553d0f64bfebc9ac85f99a54163503b1e3b9e95e158e77193bca62d315"}
Jul 28 01:27:15.101217 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.101139    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh" event={"ID":"2ac8110e-26ae-412f-8188-bd530bf3f0d4","Type":"ContainerStarted","Data":"1c8d3f7990a60768ebf416354b24c4c9f9ded91070f3c4f9d72ce52b2ae94095"}
Jul 28 01:27:15.105215 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.104596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j" event={"ID":"a780f93f-13ea-4bd5-b370-64c18d4e5c60","Type":"ContainerStarted","Data":"a5d1e7f514f592d0b38071753210fc1e562eaede6a2d23aa0dccb8e2dad97ca4"}
Jul 28 01:27:15.106537 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.106459    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:27:15.110524 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.110427    2581 generic.go:356] "Generic (PLEG): container finished" podID="f1c85108-418d-4c11-8762-8231d4f84826" containerID="c54da5e32095b2626542db854795251dd767209be759099905a6731419e5b618" exitCode=0
Jul 28 01:27:15.110666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.110540    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qq55" event={"ID":"f1c85108-418d-4c11-8762-8231d4f84826","Type":"ContainerDied","Data":"c54da5e32095b2626542db854795251dd767209be759099905a6731419e5b618"}
Jul 28 01:27:15.114295 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.114235    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j"
Jul 28 01:27:15.128346 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.128254    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vbj2j" podStartSLOduration=7.49024965 podStartE2EDuration="18.128237495s" totalImagesPullingTime="10.637987845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:15.127893867 +0000 UTC m=+84.033463266" watchObservedRunningTime="2026-07-28 01:27:15.128237495 +0000 UTC m=+84.033806892"
Jul 28 01:27:15.135900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.135771    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4" event={"ID":"c0dd6710-e852-46bb-991c-cac90d69c12d","Type":"ContainerStarted","Data":"61f5aa43f12164aa98e65164a5b2c0f58714f2186b5284e9f5054002c5d0734c"}
Jul 28 01:27:15.135900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.135858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4" event={"ID":"c0dd6710-e852-46bb-991c-cac90d69c12d","Type":"ContainerStarted","Data":"2afe78025c64154c8c0143adb3692fdee58444fc0c24568975b6469dd986f0ec"}
Jul 28 01:27:15.139590 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.139328    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g79hr" event={"ID":"c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7","Type":"ContainerStarted","Data":"80705fb0c28e69d474de36728dc574548148e52e6dac828b0b70336df9c7511a"}
Jul 28 01:27:15.141484 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.141457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hsjxk" event={"ID":"9d8f5494-05fc-48a7-a8c2-cbfbf30134fb","Type":"ContainerStarted","Data":"2528af8900a3639267ba00aa37af43c2bff1803fe1713bd087f89ac61ab389ab"}
Jul 28 01:27:15.148453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.148353    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst" event={"ID":"3ed4e918-20c2-4293-acb8-d32ada84f2eb","Type":"ContainerStarted","Data":"72422e9a6a260d41f2a04372609cf26a8d65951fc3938e2c1b0d9bbbb73c9e24"}
Jul 28 01:27:15.156054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.155581    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2" exitCode=0
Jul 28 01:27:15.156054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.155624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"}
Jul 28 01:27:15.216989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.216860    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-jxxst" podStartSLOduration=3.805918589 podStartE2EDuration="22.216841719s" totalImagesPullingTime="18.41092313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:15.21605385 +0000 UTC m=+84.121623257" watchObservedRunningTime="2026-07-28 01:27:15.216841719 +0000 UTC m=+84.122411118"
Jul 28 01:27:15.217885 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:15.217823    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g79hr" podStartSLOduration=73.572331025 podStartE2EDuration="1m24.217788302s" totalImagesPullingTime="10.645457277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:15.191549641 +0000 UTC m=+84.097119037" watchObservedRunningTime="2026-07-28 01:27:15.217788302 +0000 UTC m=+84.123357715"
Jul 28 01:27:16.164018 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.163974    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qq55" event={"ID":"f1c85108-418d-4c11-8762-8231d4f84826","Type":"ContainerStarted","Data":"c1bc2b0a07f5d803416238feff40b564e5e5b54c39b5e98d4b75593ec8d4d2d9"}
Jul 28 01:27:16.164569 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.164046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2qq55" event={"ID":"f1c85108-418d-4c11-8762-8231d4f84826","Type":"ContainerStarted","Data":"9a2aff9c16813746d3db5350b8f1929b7735d63154fcf0121b5dceed20eaff08"}
Jul 28 01:27:16.167748 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.167261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4" event={"ID":"c0dd6710-e852-46bb-991c-cac90d69c12d","Type":"ContainerStarted","Data":"2c5faddf71871b02a1105731c5dbf1f9340d0900aa98ced689dd6161df1ddee9"}
Jul 28 01:27:16.169271 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.169247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tr4zt" event={"ID":"329df437-fb66-4775-a16e-94b06ef0cf80","Type":"ContainerStarted","Data":"4129a21f06e491bfca947d3906c173e467b164eaeef98a02f0d4843d6d25dabb"}
Jul 28 01:27:16.171950 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.171921    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh" event={"ID":"2ac8110e-26ae-412f-8188-bd530bf3f0d4","Type":"ContainerStarted","Data":"b7264af5ab17a2b144c710dabfecdcf38793cd7f623922435d6ea877b8f70827"}
Jul 28 01:27:16.172093 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.171980    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh" event={"ID":"2ac8110e-26ae-412f-8188-bd530bf3f0d4","Type":"ContainerStarted","Data":"10bc2f7b179974bb347aa160f0f30be44b635f19a8796667ebed24e1aad55192"}
Jul 28 01:27:16.172579 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.172533    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:27:16.187559 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.187503    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2qq55" podStartSLOduration=4.352005438 podStartE2EDuration="23.187485383s" totalImagesPullingTime="18.718061434s" totalInitContainerRuntime="117.418511ms" isStatefulPod=true podCreationTimestamp="2026-07-28 01:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.185282118 +0000 UTC m=+85.090851542" watchObservedRunningTime="2026-07-28 01:27:16.187485383 +0000 UTC m=+85.093054833"
Jul 28 01:27:16.188142 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.188103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hsjxk" podStartSLOduration=74.547989826 podStartE2EDuration="1m25.188092755s" totalImagesPullingTime="10.640102929s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:15.264915239 +0000 UTC m=+84.170484638" watchObservedRunningTime="2026-07-28 01:27:16.188092755 +0000 UTC m=+85.093662131"
Jul 28 01:27:16.213535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.213450    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7fc868b465-cpngh" podStartSLOduration=7.570699412 podStartE2EDuration="18.213425449s" totalImagesPullingTime="10.642726037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.212538192 +0000 UTC m=+85.118107594" watchObservedRunningTime="2026-07-28 01:27:16.213425449 +0000 UTC m=+85.118994849"
Jul 28 01:27:16.233815 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.233756    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tr4zt" podStartSLOduration=74.432181572 podStartE2EDuration="1m25.233739943s" totalImagesPullingTime="10.801558371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.232778172 +0000 UTC m=+85.138347592" watchObservedRunningTime="2026-07-28 01:27:16.233739943 +0000 UTC m=+85.139309340"
Jul 28 01:27:16.251330 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.251278    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-lmxt4" podStartSLOduration=4.74323966 podStartE2EDuration="23.251257685s" totalImagesPullingTime="18.508018025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:16.250514489 +0000 UTC m=+85.156083889" watchObservedRunningTime="2026-07-28 01:27:16.251257685 +0000 UTC m=+85.156827085"
Jul 28 01:27:16.379953 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.379916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:16.477534 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:16.477452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.322281 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:17.322243    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:17.514396 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:17.514361    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:20.197034 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:20.196951    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"}
Jul 28 01:27:20.197034 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:20.197015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"}
Jul 28 01:27:20.331126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:20.331096    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmhz4_962dc654-599f-420d-979a-95ce484e7b2a/serve-healthcheck-canary/0.log"
Jul 28 01:27:21.386869 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:21.386830    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:21.455728 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:21.455697    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:23.211355 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:23.211310    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"}
Jul 28 01:27:23.211953 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:23.211375    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"}
Jul 28 01:27:23.211953 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:23.211402    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"}
Jul 28 01:27:23.211953 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:23.211426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerStarted","Data":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"}
Jul 28 01:27:23.281518 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:23.281456    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=6.723924796 podStartE2EDuration="24.281434817s" totalImagesPullingTime="17.5414355s" totalInitContainerRuntime="16.074521ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:26:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:27:23.279524955 +0000 UTC m=+92.185094366" watchObservedRunningTime="2026-07-28 01:27:23.281434817 +0000 UTC m=+92.187004215"
Jul 28 01:27:24.698646 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:24.698606    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:28.964477 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:28.964438    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.065684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:29.065647    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:29.149326 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:29.149294    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:35.774539 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:35.774481    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74665dbc9f-4mrgl" podUID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" containerName="console" containerID="cri-o://e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322" gracePeriod=15
Jul 28 01:27:36.063021 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.062987    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:36.137984 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.137941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.137984 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.137988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.138237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138028    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.138237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138066    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.138237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.138237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138156    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config\") pod \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\" (UID: \"74ddfd47-339a-46d8-bdc9-8d6a8a26434b\") "
Jul 28 01:27:36.138500 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca" (OuterVolumeSpecName: "service-ca") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:36.138582 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138495    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:36.138582 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.138481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config" (OuterVolumeSpecName: "console-config") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:27:36.140420 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.140385    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:36.140535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.140433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs" (OuterVolumeSpecName: "kube-api-access-6bbqs") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "kube-api-access-6bbqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:27:36.140535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.140460    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "74ddfd47-339a-46d8-bdc9-8d6a8a26434b" (UID: "74ddfd47-339a-46d8-bdc9-8d6a8a26434b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:27:36.238962 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.238916    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-oauth-serving-cert\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.238962 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.238954    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-oauth-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.238962 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.238967    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bbqs\" (UniqueName: \"kubernetes.io/projected/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-kube-api-access-6bbqs\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.239211 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.238979    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-service-ca\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.239211 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.238991    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-serving-cert\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.239211 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.239004    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/74ddfd47-339a-46d8-bdc9-8d6a8a26434b-console-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:27:36.266401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.266361    2581 generic.go:356] "Generic (PLEG): container finished" podID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" containerID="e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322" exitCode=0
Jul 28 01:27:36.266586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.266421    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74665dbc9f-4mrgl"
Jul 28 01:27:36.266586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.266451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74665dbc9f-4mrgl" event={"ID":"74ddfd47-339a-46d8-bdc9-8d6a8a26434b","Type":"ContainerDied","Data":"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"}
Jul 28 01:27:36.266586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.266498    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74665dbc9f-4mrgl" event={"ID":"74ddfd47-339a-46d8-bdc9-8d6a8a26434b","Type":"ContainerDied","Data":"da30d3e60b4f9b67b4f1323fd89fb072b89fa090e62db8795d7257e9fabe8438"}
Jul 28 01:27:36.266586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.266518    2581 scope.go:117] "RemoveContainer" containerID="e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"
Jul 28 01:27:36.275671 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.275655    2581 scope.go:117] "RemoveContainer" containerID="e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"
Jul 28 01:27:36.275970 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:27:36.275951    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322\": container with ID starting with e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322 not found: ID does not exist" containerID="e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"
Jul 28 01:27:36.276030 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.275980    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322"} err="failed to get container status \"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322\": rpc error: code = NotFound desc = could not find container \"e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322\": container with ID starting with e660f44ea47c55db437a4c72e683e3859d9dc046e66e396e01d84d4d606c4322 not found: ID does not exist"
Jul 28 01:27:36.286341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.286260    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:27:36.292341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:36.292310    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74665dbc9f-4mrgl"]
Jul 28 01:27:37.647046 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:37.647014    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" path="/var/lib/kubelet/pods/74ddfd47-339a-46d8-bdc9-8d6a8a26434b/volumes"
Jul 28 01:27:37.693880 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:37.693847    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.815222 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:37.815190    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:37.884897 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:37.884867    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:27:47.182472 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:47.182439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g79hr"
Jul 28 01:27:59.698549 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:59.698508    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:27:59.763391 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:27:59.763361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:00.370860 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:00.370830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:09.211753 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:09.211712    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:09.274440 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:09.274406    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:17.714918 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.714878    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715196    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="prometheus" containerID="cri-o://83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5" gracePeriod=600
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715216    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0" gracePeriod=600
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715245    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-web" containerID="cri-o://6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643" gracePeriod=600
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715242    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="config-reloader" containerID="cri-o://dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e" gracePeriod=600
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715216    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy" containerID="cri-o://231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02" gracePeriod=600
Jul 28 01:28:17.715475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.715253    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="thanos-sidecar" containerID="cri-o://2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7" gracePeriod=600
Jul 28 01:28:17.985262 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:17.985234    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.011822 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.011768    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.011839    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.011876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.011902    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-995jm\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012022    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012099    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012129    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012158    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012189    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012218    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012252    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012298    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012344    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012345    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012368    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012436    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012462    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012489    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.012515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012520    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2\") pod \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\" (UID: \"4a8ffc98-be42-4cc0-b55d-e483adbd9b55\") "
Jul 28 01:28:18.013072 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.012771    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.013188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.013160    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.015020 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.013469    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.015020 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.013787    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.015643 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.015531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.016099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.016065    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.016602 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.016569    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.016602 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.016583    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:28:18.017364 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.017273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.018988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.019054    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out" (OuterVolumeSpecName: "config-out") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.019113    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.020222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm" (OuterVolumeSpecName: "kube-api-access-995jm") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "kube-api-access-995jm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.020575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.020945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.022321    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.022507    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.024011    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.024538 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.024284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config" (OuterVolumeSpecName: "config") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.035592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.035547    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config" (OuterVolumeSpecName: "web-config") pod "4a8ffc98-be42-4cc0-b55d-e483adbd9b55" (UID: "4a8ffc98-be42-4cc0-b55d-e483adbd9b55"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:28:18.114099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114053    2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114090    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114101    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114110    2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-kube-rbac-proxy\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114119    2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-web-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114128    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-metrics-client-ca\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114137    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114145    2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-metrics-client-certs\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114155    2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-config-out\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114163    2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-thanos-prometheus-http-client-file\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114172    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-995jm\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-kube-api-access-995jm\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114180    2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-tls-assets\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114188    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114196    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-trusted-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114204    2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-prometheus-k8s-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114212    2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114222    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114230    2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-secret-grpc-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.114371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.114237    2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4a8ffc98-be42-4cc0-b55d-e483adbd9b55-prometheus-k8s-db\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:28:18.423692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423650    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0" exitCode=0
Jul 28 01:28:18.423692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423680    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02" exitCode=0
Jul 28 01:28:18.423692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423689    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643" exitCode=0
Jul 28 01:28:18.423692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423695    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7" exitCode=0
Jul 28 01:28:18.423692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423702    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e" exitCode=0
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423708    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5" exitCode=0
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423700    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423738    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423872    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423878    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"}
Jul 28 01:28:18.424036 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.423949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4a8ffc98-be42-4cc0-b55d-e483adbd9b55","Type":"ContainerDied","Data":"a9ab821047765f87d359e6fffd29c8238faa58bceda3c394fa02a99ebe869392"}
Jul 28 01:28:18.432439 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.432418    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.439604 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.439581    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.447049 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.447025    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.447956 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.447930    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:18.454523 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.454499    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:18.454868 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.454783    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.461660 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.461639    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.469196 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.469176    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.476388 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.476370    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.476667 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.476648    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.476712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.476678    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.476712 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.476698    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.477006 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.476979    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.477110 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477012    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.477110 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477034    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.477305 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.477290    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.477352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477313    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.477352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477328    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.477552 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.477535    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.477603 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477557    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.477603 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477569    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.477788 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.477759    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.477899 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477785    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.477899 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.477819    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.478191 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.478166    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.478303 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478195    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.478303 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478211    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.478824 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:28:18.478507    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.478824 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478534    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.478824 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478553    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.479030 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478870    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.479030 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.478892    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.479168 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479129    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.479207 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479171    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.479423 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479404    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.479470 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479425    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.479673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479651    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.479761 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479676    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.479945 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479930    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.480005 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.479946    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.480163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480138    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.480163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480154    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.480251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480176    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:18.480351 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480333    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.480393 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480351    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.480495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480479    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy"
Jul 28 01:28:18.480531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480498    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy"
Jul 28 01:28:18.480531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480518    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="init-config-reloader"
Jul 28 01:28:18.480531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480528    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="init-config-reloader"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480537    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-thanos"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480537    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480562    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480543    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-thanos"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480579    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="prometheus"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480585    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="prometheus"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480596    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="config-reloader"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480601    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="config-reloader"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480611    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="thanos-sidecar"
Jul 28 01:28:18.480618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480618    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="thanos-sidecar"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480626    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" containerName="console"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480633    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" containerName="console"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480639    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-web"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480646    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-web"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480698    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="config-reloader"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480712    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480721    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-thanos"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480730    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="thanos-sidecar"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480738    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="kube-rbac-proxy-web"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480747    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="74ddfd47-339a-46d8-bdc9-8d6a8a26434b" containerName="console"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" containerName="prometheus"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480831    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.481010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.480847    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.481488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481086    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.481488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481110    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.481488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481317    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.481488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481333    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.481632 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481584    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.481632 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481601    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.481849 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481833    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.481896 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.481850    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.482096 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482079    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.482096 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482095    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.482317 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482300    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.482352 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482318    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.482536 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482522    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.482576 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482536    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.482728 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482711    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.482779 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482728    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.482947 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482929    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.483016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.482949    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.483171 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483152    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.483210 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483172    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.483404 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483387    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.483447 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483406    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.483605 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483589    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.483645 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483605    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.483774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483755    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.483774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.483774    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.484054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484036    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.484098 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484054    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.484281 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484264    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.484334 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484282    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.484471 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484452    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.484525 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484471    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.484889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484872    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.484961 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.484889    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.485260 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.485187    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.485260 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.485213    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.485587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.485492    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.485587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.485516    2581 scope.go:117] "RemoveContainer" containerID="1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"
Jul 28 01:28:18.485587 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.485559    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.487175 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487147    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0"} err="failed to get container status \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": rpc error: code = NotFound desc = could not find container \"1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0\": container with ID starting with 1d187a17af49ac0097cc23f5c7848f8850e0325bd13f1de013866ec59bf316f0 not found: ID does not exist"
Jul 28 01:28:18.487175 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487173    2581 scope.go:117] "RemoveContainer" containerID="231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"
Jul 28 01:28:18.487530 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487507    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02"} err="failed to get container status \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": rpc error: code = NotFound desc = could not find container \"231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02\": container with ID starting with 231710f5f16cef76072d4693a3bc47e26d9030d88efb05320728610747b15c02 not found: ID does not exist"
Jul 28 01:28:18.487603 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487532    2581 scope.go:117] "RemoveContainer" containerID="6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"
Jul 28 01:28:18.487877 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487847    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643"} err="failed to get container status \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": rpc error: code = NotFound desc = could not find container \"6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643\": container with ID starting with 6220fee63e927a012e0575a2e403d39abd281c3de400601a1467bac111c78643 not found: ID does not exist"
Jul 28 01:28:18.487957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.487902    2581 scope.go:117] "RemoveContainer" containerID="2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"
Jul 28 01:28:18.488505 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488464    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 01:28:18.488620 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-7fd5v\""
Jul 28 01:28:18.488620 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488559    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 01:28:18.488837 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488813    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7"} err="failed to get container status \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": rpc error: code = NotFound desc = could not find container \"2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7\": container with ID starting with 2ed4fa27b8f13ebb9733f9b94ea50f6f5215f68c9f336edec2c7aee7b8659bf7 not found: ID does not exist"
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488841    2581 scope.go:117] "RemoveContainer" containerID="dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488869    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.488940    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 01:28:18.489065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489003    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 01:28:18.489356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489099    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e"} err="failed to get container status \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": rpc error: code = NotFound desc = could not find container \"dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e\": container with ID starting with dfde8ae137f4107c886eff91a0c1fa73c2be17b8594c2073e61ecaa7cefd4e3e not found: ID does not exist"
Jul 28 01:28:18.489356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489131    2581 scope.go:117] "RemoveContainer" containerID="83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"
Jul 28 01:28:18.489356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489149    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-clfv5bml756ih\""
Jul 28 01:28:18.489356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489212    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 01:28:18.489356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489255    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 01:28:18.489586 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489555    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5"} err="failed to get container status \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": rpc error: code = NotFound desc = could not find container \"83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5\": container with ID starting with 83986509ac3b551de71d647b0e2e6ef8dd544665da38ca92e08a547f267a79d5 not found: ID does not exist"
Jul 28 01:28:18.489653 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489589    2581 scope.go:117] "RemoveContainer" containerID="9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"
Jul 28 01:28:18.489653 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489629    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 01:28:18.489653 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489642    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 01:28:18.489812 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 01:28:18.489992 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.489961    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2"} err="failed to get container status \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": rpc error: code = NotFound desc = could not find container \"9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2\": container with ID starting with 9d4b822fa7aaacebcb99d2c166542d890feffeff121ae447a6dce3f131978fd2 not found: ID does not exist"
Jul 28 01:28:18.490089 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.490076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 01:28:18.492667 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.492646    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 01:28:18.497787 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.497766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 01:28:18.502588 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.502564    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:18.518127 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518127 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518129    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-config-out\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518186    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518339 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518327    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-web-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518419    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518446    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4n7f\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-kube-api-access-x4n7f\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518488    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518527    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518578    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518593    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.518861 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.518681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.619905 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.619862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.619905 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.619908    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.619929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4n7f\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-kube-api-access-x4n7f\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.619946    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.619968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620132    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620198    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620357    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620379    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-config-out\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620513    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-web-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.620910 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.620876    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.621156 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.621058    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.621298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.621271    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.623188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.622875    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.623188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.623063    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.623495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.623471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-web-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.623893 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.623870    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.623991 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.623908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624233 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624183    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-config\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624233 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624233 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624214    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/4cecabe8-0238-4a97-9ca7-ac890af12940-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624417 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624300    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624506 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624483    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.624778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.624757    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4cecabe8-0238-4a97-9ca7-ac890af12940-config-out\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.625613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.625594    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.625951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.625925    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.626279 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.626262    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.626385 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.626368    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/4cecabe8-0238-4a97-9ca7-ac890af12940-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.627866 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.627851    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4n7f\" (UniqueName: \"kubernetes.io/projected/4cecabe8-0238-4a97-9ca7-ac890af12940-kube-api-access-x4n7f\") pod \"prometheus-k8s-0\" (UID: \"4cecabe8-0238-4a97-9ca7-ac890af12940\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.797131 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.797082    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:18.943639 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:18.943599    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 01:28:18.947119 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:28:18.947089    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cecabe8_0238_4a97_9ca7_ac890af12940.slice/crio-8a81b003885ba009428830218d5cf58d411e77a1821c22b19deb161ee98e7571 WatchSource:0}: Error finding container 8a81b003885ba009428830218d5cf58d411e77a1821c22b19deb161ee98e7571: Status 404 returned error can't find the container with id 8a81b003885ba009428830218d5cf58d411e77a1821c22b19deb161ee98e7571
Jul 28 01:28:19.429411 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:19.429374    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cecabe8-0238-4a97-9ca7-ac890af12940" containerID="3441adc59345e3770d3453e35a05b6771f4325593cbf583bbf2b2aeb33f9d644" exitCode=0
Jul 28 01:28:19.429601 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:19.429459    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerDied","Data":"3441adc59345e3770d3453e35a05b6771f4325593cbf583bbf2b2aeb33f9d644"}
Jul 28 01:28:19.429601 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:19.429549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"8a81b003885ba009428830218d5cf58d411e77a1821c22b19deb161ee98e7571"}
Jul 28 01:28:19.650623 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:19.650211    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a8ffc98-be42-4cc0-b55d-e483adbd9b55" path="/var/lib/kubelet/pods/4a8ffc98-be42-4cc0-b55d-e483adbd9b55/volumes"
Jul 28 01:28:20.437028 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.436991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"ba107d41651c3c095c77467659b00cd26c53cb3356af8e2379007956ad8453ec"}
Jul 28 01:28:20.437461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.437070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"44fc17bdd64897df73406392a4cc3cc2c329b75af6e383001160cebf131af669"}
Jul 28 01:28:20.437461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.437112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"3d0670e57d512a36a8a6dd8c16280928d2718b7092a56cb4e44236ee7c2c47c9"}
Jul 28 01:28:20.437461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.437153    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"f34da94818cff29248100d9273674092d5d7143de07837d86fbd01a9309360c2"}
Jul 28 01:28:20.437461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.437196    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"cb7a599275bce43d34e3908a8a626eff3b023a6c2087222b6c2babf5226d90d5"}
Jul 28 01:28:20.437461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.437237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"4cecabe8-0238-4a97-9ca7-ac890af12940","Type":"ContainerStarted","Data":"b1b107b9897afb718c08fa7d5a7e73266ea4894052e6d205717f2f9857f5caec"}
Jul 28 01:28:20.488091 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:20.488030    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.46876864 podStartE2EDuration="2.488011517s" totalImagesPullingTime="0s" totalInitContainerRuntime="19.242877ms" isStatefulPod=false podCreationTimestamp="2026-07-28 01:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:28:20.485503114 +0000 UTC m=+149.391072524" watchObservedRunningTime="2026-07-28 01:28:20.488011517 +0000 UTC m=+149.393580915"
Jul 28 01:28:21.151783 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:21.151749    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:21.210700 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:21.210660    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:23.797559 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:23.797518    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:28:36.395876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:36.395843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:36.496767 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:36.496735    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:40.575046 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:40.575011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:40.674676 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:40.674640    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:40.729204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:40.729164    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:41.525705 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:41.525663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:41.640401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:41.640365    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:50.562251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:50.562215    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:50.622649 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:50.622606    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:51.228614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:51.228585    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:51.329406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:51.329373    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:28:51.395097 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:28:51.395065    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:29:18.798183 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:29:18.798124    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:29:18.814043 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:29:18.814016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:29:19.678256 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:29:19.678220    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 01:30:27.804737 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:27.804631    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:51.458690 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:51.458652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:30:51.460675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:51.460652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:30:51.462101 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:51.462085    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 01:30:55.125991 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:55.125945    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:55.227885 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:55.227843    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:55.497295 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:55.497187    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:55.597613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:55.597575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:30:55.697177 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:30:55.697146    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:17.987218 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:17.987171    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:18.075618 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:18.075579    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:24.601353 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:24.601315    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.013046 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.013017    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.265668 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.265576    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.365402 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.365368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.568361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.568270    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.799291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.799258    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:25.900090 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:25.899988    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:26.140374 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:26.140341    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:26.238212 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:26.238116    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:26.338646 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:26.338611    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:26.438710 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:26.438679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:26.536846 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:26.536792    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:31:51.015674 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:31:51.015575    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:41.395224 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:41.395177    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:41.495672 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:41.495632    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:43.973714 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:43.973679    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.071305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.071271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.154546 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.154510    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.264733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.264701    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.368483 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.368450    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.775987 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.775952    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:44.868342 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:44.868312    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:52.177565 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:52.177524    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:52.236466 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:52.236429    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:32:52.294042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:32:52.294011    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:33:14.852341 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:33:14.852249    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:03.214577 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.214541    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:03.256615 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.256579    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-tfpx5"]
Jul 28 01:34:03.259292 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.259271    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.267651 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.267622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 01:34:03.267824 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.267668    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 01:34:03.268453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.268432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-cpmsn\""
Jul 28 01:34:03.268537 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.268483    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 01:34:03.269218 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.269200    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-hq65f"]
Jul 28 01:34:03.271414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.271400    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.273789 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.273770    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-bjz4j\""
Jul 28 01:34:03.273789 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.273783    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 01:34:03.282784 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.282758    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tfpx5"]
Jul 28 01:34:03.285399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.285373    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-hq65f"]
Jul 28 01:34:03.325369 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.325328    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:03.380392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.380359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b83d844-05b5-4576-9a9e-71c438e011c5-cert\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.380573 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.380408    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jjl\" (UniqueName: \"kubernetes.io/projected/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-kube-api-access-85jjl\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.380573 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.380487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-tls-certs\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.380573 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.380543    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p69st\" (UniqueName: \"kubernetes.io/projected/0b83d844-05b5-4576-9a9e-71c438e011c5-kube-api-access-p69st\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.427945 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.427916    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:03.481017 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.480922    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p69st\" (UniqueName: \"kubernetes.io/projected/0b83d844-05b5-4576-9a9e-71c438e011c5-kube-api-access-p69st\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.481017 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.480989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b83d844-05b5-4576-9a9e-71c438e011c5-cert\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.481216 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.481044    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jjl\" (UniqueName: \"kubernetes.io/projected/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-kube-api-access-85jjl\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.481216 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.481096    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-tls-certs\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.483393 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.483363    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-tls-certs\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.483510 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.483444    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b83d844-05b5-4576-9a9e-71c438e011c5-cert\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.489548 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.489521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jjl\" (UniqueName: \"kubernetes.io/projected/c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a-kube-api-access-85jjl\") pod \"model-serving-api-7b954ffc95-tfpx5\" (UID: \"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a\") " pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.489658 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.489521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p69st\" (UniqueName: \"kubernetes.io/projected/0b83d844-05b5-4576-9a9e-71c438e011c5-kube-api-access-p69st\") pod \"odh-model-controller-75d9cdbddf-hq65f\" (UID: \"0b83d844-05b5-4576-9a9e-71c438e011c5\") " pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.525628 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.525592    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:03.570259 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.570227    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:03.580546 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.580509    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:03.728888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.728861    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-tfpx5"]
Jul 28 01:34:03.731424 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:34:03.731375    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc19b2a1a_a1ce_4ab6_af4b_0f4e9d15b75a.slice/crio-3c3b84349138df65b1640ef26be36f2ffa5fbbd10a8f32be557c3d9002fd4981 WatchSource:0}: Error finding container 3c3b84349138df65b1640ef26be36f2ffa5fbbd10a8f32be557c3d9002fd4981: Status 404 returned error can't find the container with id 3c3b84349138df65b1640ef26be36f2ffa5fbbd10a8f32be557c3d9002fd4981
Jul 28 01:34:03.733122 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.733106    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:34:03.740287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:03.740259    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-hq65f"]
Jul 28 01:34:03.742876 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:34:03.742849    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b83d844_05b5_4576_9a9e_71c438e011c5.slice/crio-e31549b9120d04ca8fffc45f78cc3a91eea2b116de1b989ea9c9a5745a6eb482 WatchSource:0}: Error finding container e31549b9120d04ca8fffc45f78cc3a91eea2b116de1b989ea9c9a5745a6eb482: Status 404 returned error can't find the container with id e31549b9120d04ca8fffc45f78cc3a91eea2b116de1b989ea9c9a5745a6eb482
Jul 28 01:34:04.020812 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:04.020767    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:04.120210 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:04.120179    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:34:04.720015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:04.719915    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-hq65f" event={"ID":"0b83d844-05b5-4576-9a9e-71c438e011c5","Type":"ContainerStarted","Data":"e31549b9120d04ca8fffc45f78cc3a91eea2b116de1b989ea9c9a5745a6eb482"}
Jul 28 01:34:04.721854 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:04.721754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tfpx5" event={"ID":"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a","Type":"ContainerStarted","Data":"3c3b84349138df65b1640ef26be36f2ffa5fbbd10a8f32be557c3d9002fd4981"}
Jul 28 01:34:07.735058 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.735017    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-hq65f" event={"ID":"0b83d844-05b5-4576-9a9e-71c438e011c5","Type":"ContainerStarted","Data":"a12ccea4b7dd62154fb0381cf0a31d71474b6babe9fb0e1699ce18ce4470456f"}
Jul 28 01:34:07.735523 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.735091    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:07.736696 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.736673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-tfpx5" event={"ID":"c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a","Type":"ContainerStarted","Data":"018a21e725ab1edc1c5c10688e4ae32af7c9c9bfad99a967dfcaf65f1d14209b"}
Jul 28 01:34:07.736851 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.736835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:07.756814 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.756728    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-hq65f" podStartSLOduration=1.232377384 podStartE2EDuration="4.756701873s" totalImagesPullingTime="3.524324489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:07.755040964 +0000 UTC m=+496.660610594" watchObservedRunningTime="2026-07-28 01:34:07.756701873 +0000 UTC m=+496.662271270"
Jul 28 01:34:07.770556 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:07.770494    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-tfpx5" podStartSLOduration=1.262969907 podStartE2EDuration="4.770472293s" totalImagesPullingTime="3.507502386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:07.77000067 +0000 UTC m=+496.675570049" watchObservedRunningTime="2026-07-28 01:34:07.770472293 +0000 UTC m=+496.676041682"
Jul 28 01:34:18.743108 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:18.743076    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-hq65f"
Jul 28 01:34:18.745206 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:18.745183    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-tfpx5"
Jul 28 01:34:19.587555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.587488    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-8nncc"]
Jul 28 01:34:19.593943 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.593913    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.597594 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.597476    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 01:34:19.597756 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.597477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-9497s\""
Jul 28 01:34:19.598644 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.598620    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8nncc"]
Jul 28 01:34:19.623159 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.623124    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.623366 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.623183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd6c\" (UniqueName: \"kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.724015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.723979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.724015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.724026    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd6c\" (UniqueName: \"kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.724357 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.724334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.732561 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.732527    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd6c\" (UniqueName: \"kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c\") pod \"s3-init-8nncc\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") " pod="kserve/s3-init-8nncc"
Jul 28 01:34:19.915923 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:19.915816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8nncc"
Jul 28 01:34:20.058209 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:20.058065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-8nncc"]
Jul 28 01:34:20.061101 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:34:20.061061    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c4eb5b0_a3b9_4463_9099_4c5aa33853e6.slice/crio-cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f WatchSource:0}: Error finding container cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f: Status 404 returned error can't find the container with id cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f
Jul 28 01:34:20.787079 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:20.787036    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerStarted","Data":"cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f"}
Jul 28 01:34:23.800113 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:23.800073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerStarted","Data":"ae6c58ce53f6f5d7f25f4cb6d489715f2a95108c2192e2d0500e029b46dc4125"}
Jul 28 01:34:25.810386 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:25.810349    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerID="ae6c58ce53f6f5d7f25f4cb6d489715f2a95108c2192e2d0500e029b46dc4125" exitCode=0
Jul 28 01:34:25.810818 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:25.810422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerDied","Data":"ae6c58ce53f6f5d7f25f4cb6d489715f2a95108c2192e2d0500e029b46dc4125"}
Jul 28 01:34:30.837440 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:30.837401    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerStarted","Data":"fa19c6e600ceebf7da9d5ad3dd7224617e2325866554bf006f7495f4dda8f832"}
Jul 28 01:34:30.856721 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:30.856656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-8nncc" podStartSLOduration=1.928317878 podStartE2EDuration="11.856640217s" totalImagesPullingTime="8.011544097s" totalInitContainerRuntime="1.916778242s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:34:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:34:30.855021822 +0000 UTC m=+519.760591215" watchObservedRunningTime="2026-07-28 01:34:30.856640217 +0000 UTC m=+519.762209680"
Jul 28 01:34:50.263553 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:34:50.263464    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:29.947493 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:29.947451    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:36.088670 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:36.088634    2581 generic.go:356] "Generic (PLEG): container finished" podID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerID="fa19c6e600ceebf7da9d5ad3dd7224617e2325866554bf006f7495f4dda8f832" exitCode=0
Jul 28 01:35:36.089133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:36.088708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerDied","Data":"fa19c6e600ceebf7da9d5ad3dd7224617e2325866554bf006f7495f4dda8f832"}
Jul 28 01:35:37.219592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.219563    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8nncc"
Jul 28 01:35:37.284724 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.284685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkd6c\" (UniqueName: \"kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c\") pod \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") "
Jul 28 01:35:37.284949 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.284818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache\") pod \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\" (UID: \"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6\") "
Jul 28 01:35:37.286983 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.286949    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c" (OuterVolumeSpecName: "kube-api-access-bkd6c") pod "3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" (UID: "3c4eb5b0-a3b9-4463-9099-4c5aa33853e6"). InnerVolumeSpecName "kube-api-access-bkd6c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:35:37.304416 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.304375    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache" (OuterVolumeSpecName: "model-cache") pod "3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" (UID: "3c4eb5b0-a3b9-4463-9099-4c5aa33853e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:35:37.385544 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.385464    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkd6c\" (UniqueName: \"kubernetes.io/projected/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-kube-api-access-bkd6c\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:35:37.385544 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:37.385495    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c4eb5b0-a3b9-4463-9099-4c5aa33853e6-model-cache\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:35:38.099078 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:38.099046    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-8nncc"
Jul 28 01:35:38.099283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:38.099045    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-8nncc" event={"ID":"3c4eb5b0-a3b9-4463-9099-4c5aa33853e6","Type":"ContainerDied","Data":"cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f"}
Jul 28 01:35:38.099283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:38.099157    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc620c38bf684002e1d8a03e4b113a4e2fbeb16e317496b11ecf8225f979416f"
Jul 28 01:35:38.464041 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:38.463953    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:38.566838 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:38.566786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.231266 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.231229    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.286706 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.286674    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:35:50.287006 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.286993    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerName="s3-init"
Jul 28 01:35:50.287050 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.287007    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerName="s3-init"
Jul 28 01:35:50.287050 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.287019    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerName="download-hf-model"
Jul 28 01:35:50.287050 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.287025    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerName="download-hf-model"
Jul 28 01:35:50.287144 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.287078    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c4eb5b0-a3b9-4463-9099-4c5aa33853e6" containerName="s3-init"
Jul 28 01:35:50.289378 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.289360    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.291623 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.291577    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7854-predictor-serving-cert\""
Jul 28 01:35:50.291791 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.291766    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7854-kube-rbac-proxy-sar-config\""
Jul 28 01:35:50.291791 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.291768    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 01:35:50.291952 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.291887    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 01:35:50.292613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.292595    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-mcfll\""
Jul 28 01:35:50.301875 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.301852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:35:50.399873 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.399832    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hswd\" (UniqueName: \"kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.400049 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.399914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.400049 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.399938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.419243 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.419195    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.501316 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.501292    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hswd\" (UniqueName: \"kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.501488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.501363    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.501488 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.501394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.502248 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.502209    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.503882 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.503862    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.513765 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.513743    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hswd\" (UniqueName: \"kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd\") pod \"success-200-isvc-c7854-predictor-644bcbbbf7-7k64h\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") " pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.535719 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.535682    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.578683 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.578646    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:35:50.582078 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.582056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.584290 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.584257    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 01:35:50.584436 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.584317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 01:35:50.584436 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.584347    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-predictor-serving-cert\""
Jul 28 01:35:50.584436 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.584317    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-1-kube-rbac-proxy-sar-config\""
Jul 28 01:35:50.591175 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.591148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:35:50.604272 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.604241    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:35:50.636819 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.636763    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.668853 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.668822    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:35:50.672345 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.672317    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.675157 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.674720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-predictor-serving-cert\""
Jul 28 01:35:50.675157 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.675038    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-kube-rbac-proxy-sar-config\""
Jul 28 01:35:50.682188 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.681994    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:35:50.703138 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.703102    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.703305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.703160    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.703305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.703201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8hmh\" (UniqueName: \"kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.703305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.703283    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.703475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.703331    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.735562 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.735530    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.754900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.754841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:35:50.758223 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:35:50.758193    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdec550_0d4b_453e_ac45_84fea7a86bd2.slice/crio-a5d63978d05867591edb9fe516e23047e1660f471d0e11d429a117a8d96987a7 WatchSource:0}: Error finding container a5d63978d05867591edb9fe516e23047e1660f471d0e11d429a117a8d96987a7: Status 404 returned error can't find the container with id a5d63978d05867591edb9fe516e23047e1660f471d0e11d429a117a8d96987a7
Jul 28 01:35:50.804251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804222    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804329    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5wr\" (UniqueName: \"kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8hmh\" (UniqueName: \"kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.804445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804424    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.804834 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804461    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.804834 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:35:50.804474    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-serving-cert: secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 01:35:50.804834 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.804492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.804834 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:35:50.804548    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls podName:ffa80abb-6239-4e15-b294-cfde80f6becf nodeName:}" failed. No retries permitted until 2026-07-28 01:35:51.304526755 +0000 UTC m=+600.210096137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls") pod "isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf") : secret "isvc-sklearn-graph-1-predictor-serving-cert" not found
Jul 28 01:35:50.805064 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.805020    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.805422 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.805400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.805422 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.805411    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.817282 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.817251    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8hmh\" (UniqueName: \"kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:50.835273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.835233    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.878044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.878013    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:35:50.882350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.882326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:50.885613 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.885593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-predictor-serving-cert\""
Jul 28 01:35:50.885725 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.885707    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-2-kube-rbac-proxy-sar-config\""
Jul 28 01:35:50.893718 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.893685    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:35:50.905771 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.905741    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.905957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.905782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.905957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.905850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5wr\" (UniqueName: \"kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.905957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.905886    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.905957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.905913    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.906246 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.906223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.906540 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.906515    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.906648 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.906598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.908313 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.908286    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.914076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.914055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5wr\" (UniqueName: \"kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr\") pod \"isvc-xgboost-graph-predictor-555fc69d56-b2mj6\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:50.922177 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.922150    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:50.987630 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:50.987593    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:35:51.004337 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.004305    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:51.007290 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.007236    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.007290 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.007275    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.007426 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.007365    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.007426 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.007396    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7zbs\" (UniqueName: \"kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.007490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.007442    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.094101 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.094069    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:51.108515 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.108709 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7zbs\" (UniqueName: \"kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.108709 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.108709 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.108889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108713    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.108970 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.108950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.109329 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.109308    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.109329 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.109325    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.111053 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.111030    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.120217 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.120184    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7zbs\" (UniqueName: \"kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs\") pod \"isvc-sklearn-graph-2-predictor-7d44765f95-zc72k\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.126308 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.126287    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:35:51.128842 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:35:51.128817    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd23e5073_7566_4811_9ffc_96d78ea1d092.slice/crio-12d93e703c0dae34a3953396d1bcc9eb982146e979e8c727b9cfd912ab3028d5 WatchSource:0}: Error finding container 12d93e703c0dae34a3953396d1bcc9eb982146e979e8c727b9cfd912ab3028d5: Status 404 returned error can't find the container with id 12d93e703c0dae34a3953396d1bcc9eb982146e979e8c727b9cfd912ab3028d5
Jul 28 01:35:51.154387 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.154352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerStarted","Data":"12d93e703c0dae34a3953396d1bcc9eb982146e979e8c727b9cfd912ab3028d5"}
Jul 28 01:35:51.155919 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.155887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerStarted","Data":"a5d63978d05867591edb9fe516e23047e1660f471d0e11d429a117a8d96987a7"}
Jul 28 01:35:51.193135 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.193102    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:51.193931 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.193907    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:35:51.310461 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.310421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:51.313071 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.313047    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") pod \"isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:51.332365 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.332337    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:35:51.335232 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:35:51.335206    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87fd680b_b995_49c6_ad0d_62c45a833494.slice/crio-f935591f3df571a3b8d22a41970d2f21527ab7177c94ee8a87ca6824da1c4e7d WatchSource:0}: Error finding container f935591f3df571a3b8d22a41970d2f21527ab7177c94ee8a87ca6824da1c4e7d: Status 404 returned error can't find the container with id f935591f3df571a3b8d22a41970d2f21527ab7177c94ee8a87ca6824da1c4e7d
Jul 28 01:35:51.498445 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.498114    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:35:51.564174 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.564146    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:35:51.566836 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.566356    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:35:51.848737 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:51.848611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:35:51.859555 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:35:51.853067    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa80abb_6239_4e15_b294_cfde80f6becf.slice/crio-0fd5c40636713e9c66a700f6a1dbf17d699a837dff8515a2e9d1141cc63c91b6 WatchSource:0}: Error finding container 0fd5c40636713e9c66a700f6a1dbf17d699a837dff8515a2e9d1141cc63c91b6: Status 404 returned error can't find the container with id 0fd5c40636713e9c66a700f6a1dbf17d699a837dff8515a2e9d1141cc63c91b6
Jul 28 01:35:52.175431 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:52.175342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerStarted","Data":"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"}
Jul 28 01:35:52.175431 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:52.175420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerStarted","Data":"f935591f3df571a3b8d22a41970d2f21527ab7177c94ee8a87ca6824da1c4e7d"}
Jul 28 01:35:52.182544 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:52.181882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerStarted","Data":"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"}
Jul 28 01:35:52.182544 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:52.181958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerStarted","Data":"0fd5c40636713e9c66a700f6a1dbf17d699a837dff8515a2e9d1141cc63c91b6"}
Jul 28 01:35:52.186029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:52.185205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerStarted","Data":"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"}
Jul 28 01:35:56.242318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:56.242286    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:56.310005 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:56.309963    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:35:57.213082 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.213046    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerID="e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b" exitCode=0
Jul 28 01:35:57.213271 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.213132    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerDied","Data":"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"}
Jul 28 01:35:57.215020 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.214976    2581 generic.go:356] "Generic (PLEG): container finished" podID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerID="6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a" exitCode=0
Jul 28 01:35:57.215093 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.215053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerDied","Data":"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"}
Jul 28 01:35:57.217155 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.217082    2581 generic.go:356] "Generic (PLEG): container finished" podID="87fd680b-b995-49c6-ad0d-62c45a833494" containerID="39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d" exitCode=0
Jul 28 01:35:57.217155 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:57.217150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerDied","Data":"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"}
Jul 28 01:35:58.771675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:35:58.771633    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:36:08.276970 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:08.276938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerStarted","Data":"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"}
Jul 28 01:36:09.287451 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:09.287209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerStarted","Data":"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"}
Jul 28 01:36:09.290133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:09.290098    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerStarted","Data":"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"}
Jul 28 01:36:13.320492 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.320448    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerStarted","Data":"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"}
Jul 28 01:36:13.321005 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.320786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:36:13.323222 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.323185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerStarted","Data":"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"}
Jul 28 01:36:13.323377 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.323258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:36:13.323377 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.323293    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:36:13.324614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.324550    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:13.325413 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.325387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerStarted","Data":"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"}
Jul 28 01:36:13.325653 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.325623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:36:13.325778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.325757    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:36:13.326743 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.326718    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:13.340594 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.340523    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podStartSLOduration=3.633255725 podStartE2EDuration="23.340500258s" totalImagesPullingTime="15.464994321s" totalInitContainerRuntime="4.242250212s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:13.338863665 +0000 UTC m=+622.244433068" watchObservedRunningTime="2026-07-28 01:36:13.340500258 +0000 UTC m=+622.246069656"
Jul 28 01:36:13.355766 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.355708    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podStartSLOduration=1.418221655 podStartE2EDuration="23.355691297s" totalImagesPullingTime="21.937469642s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:13.354151959 +0000 UTC m=+622.259721357" watchObservedRunningTime="2026-07-28 01:36:13.355691297 +0000 UTC m=+622.261260776"
Jul 28 01:36:13.371527 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:13.371460    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podStartSLOduration=3.631686668 podStartE2EDuration="23.371436653s" totalImagesPullingTime="15.517485713s" totalInitContainerRuntime="4.222264272s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:13.37026157 +0000 UTC m=+622.275830969" watchObservedRunningTime="2026-07-28 01:36:13.371436653 +0000 UTC m=+622.277006056"
Jul 28 01:36:14.330418 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:14.330375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:14.330906 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:14.330835    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:36:14.331006 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:14.330973    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:14.331410 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:14.331386    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:15.336970 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:15.336630    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:19.337361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:19.336578    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:36:19.337361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:19.336685    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:36:19.337907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:19.337493    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:19.337907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:19.337751    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:20.343499 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:20.343464    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:36:20.344163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:20.344133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:29.337831 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:29.337471    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:29.338612 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:29.337471    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:30.344671 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.344591    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:30.413035 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.413002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerStarted","Data":"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"}
Jul 28 01:36:30.413202 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.413091    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerStarted","Data":"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"}
Jul 28 01:36:30.413386 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.413369    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:36:30.413556 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.413537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:36:30.414676 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.414645    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:36:30.437939 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:30.437852    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podStartSLOduration=3.424812637 podStartE2EDuration="40.437828075s" totalImagesPullingTime="32.869071659s" totalInitContainerRuntime="4.143943779s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:35:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:36:30.434392283 +0000 UTC m=+639.339961682" watchObservedRunningTime="2026-07-28 01:36:30.437828075 +0000 UTC m=+639.343397476"
Jul 28 01:36:31.417759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:31.417719    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:36:36.422429 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:36.422402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:36:36.423084 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:36.423056    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:36:39.337848 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:39.337789    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:39.338343 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:39.337861    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:40.344938 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:40.344901    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:46.423254 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:46.423214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:36:49.337648 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:49.337602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:49.338174 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:49.337602    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:36:50.344476 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:50.344436    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:36:56.423242 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:56.423200    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:36:59.337654 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:59.337606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 01:36:59.338173 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:36:59.337606    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:37:00.344192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:00.344157    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:37:06.423369 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:06.423332    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:37:09.337386 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:09.337334    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:37:09.337955 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:09.337936    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:37:10.344696 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:10.344655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:37:16.423994 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:16.423951    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:37:19.338299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:19.338240    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:37:20.344145 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:20.344106    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:37:20.348072 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:20.348045    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:20.665011 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:20.664920    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:26.423603 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:26.423560    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 01:37:29.338043 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:29.337993    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 01:37:30.344845 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.344780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 01:37:30.578102 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.578070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:30.702519 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.702422    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:37:30.702699 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.702673    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container" containerID="cri-o://4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688" gracePeriod=30
Jul 28 01:37:30.702772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.702704    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kube-rbac-proxy" containerID="cri-o://63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493" gracePeriod=30
Jul 28 01:37:30.727786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.727747    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:30.759499 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.759464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:37:30.762350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.762331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.764490 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.764459    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-33843-predictor-serving-cert\""
Jul 28 01:37:30.764761 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.764741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-33843-kube-rbac-proxy-sar-config\""
Jul 28 01:37:30.775031 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.774954    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:37:30.805745 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.805707    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:30.829123 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.829093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.829285 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.829148    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gg2l\" (UniqueName: \"kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.829285 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.829262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.893317 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.893280    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:30.929632 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.929591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.929879 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.929654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.929879 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.929701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gg2l\" (UniqueName: \"kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.929879 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:37:30.929758    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-33843-predictor-serving-cert: secret "success-200-isvc-33843-predictor-serving-cert" not found
Jul 28 01:37:30.929879 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:37:30.929878    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls podName:ffacdfd9-fb25-4a70-b282-36051a7dcafc nodeName:}" failed. No retries permitted until 2026-07-28 01:37:31.429855474 +0000 UTC m=+700.335424851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls") pod "success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" (UID: "ffacdfd9-fb25-4a70-b282-36051a7dcafc") : secret "success-200-isvc-33843-predictor-serving-cert" not found
Jul 28 01:37:30.930299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.930279    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.939162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.939119    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gg2l\" (UniqueName: \"kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:30.962675 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:30.962587    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:31.121243 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.121206    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:31.222370 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.222271    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:31.434162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.434121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:31.437475 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.437442    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") pod \"success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") " pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:31.674732 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.674695    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:31.677521 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.677490    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerID="63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493" exitCode=2
Jul 28 01:37:31.677631 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.677567    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerDied","Data":"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"}
Jul 28 01:37:31.815904 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:31.815878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:37:31.818173 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:37:31.818147    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffacdfd9_fb25_4a70_b282_36051a7dcafc.slice/crio-8151cc1c5c4aba59beb8113c5c4e4b36da484ce998acd710a86825d120184d8d WatchSource:0}: Error finding container 8151cc1c5c4aba59beb8113c5c4e4b36da484ce998acd710a86825d120184d8d: Status 404 returned error can't find the container with id 8151cc1c5c4aba59beb8113c5c4e4b36da484ce998acd710a86825d120184d8d
Jul 28 01:37:32.683251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.683204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerStarted","Data":"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"}
Jul 28 01:37:32.683710 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.683268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerStarted","Data":"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"}
Jul 28 01:37:32.683710 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.683288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerStarted","Data":"8151cc1c5c4aba59beb8113c5c4e4b36da484ce998acd710a86825d120184d8d"}
Jul 28 01:37:32.683710 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.683427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:32.683710 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.683480    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:32.684907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.684878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:37:32.701119 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:32.701072    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podStartSLOduration=2.701054841 podStartE2EDuration="2.701054841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:37:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:37:32.700283247 +0000 UTC m=+701.605852658" watchObservedRunningTime="2026-07-28 01:37:32.701054841 +0000 UTC m=+701.606624238"
Jul 28 01:37:33.687119 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:33.687087    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:37:34.330867 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.330822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 01:37:34.466705 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.466674    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:37:34.563991 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.563957    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls\") pod \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") "
Jul 28 01:37:34.564196 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.564005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hswd\" (UniqueName: \"kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd\") pod \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") "
Jul 28 01:37:34.564196 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.564070    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config\") pod \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\" (UID: \"5bdec550-0d4b-453e-ac45-84fea7a86bd2\") "
Jul 28 01:37:34.564481 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.564452    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5bdec550-0d4b-453e-ac45-84fea7a86bd2" (UID: "5bdec550-0d4b-453e-ac45-84fea7a86bd2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:37:34.566309 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.566278    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd" (OuterVolumeSpecName: "kube-api-access-5hswd") pod "5bdec550-0d4b-453e-ac45-84fea7a86bd2" (UID: "5bdec550-0d4b-453e-ac45-84fea7a86bd2"). InnerVolumeSpecName "kube-api-access-5hswd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:37:34.566309 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.566299    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5bdec550-0d4b-453e-ac45-84fea7a86bd2" (UID: "5bdec550-0d4b-453e-ac45-84fea7a86bd2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:37:34.665592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.665563    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.665592 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.665590    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5bdec550-0d4b-453e-ac45-84fea7a86bd2-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.665792 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.665603    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hswd\" (UniqueName: \"kubernetes.io/projected/5bdec550-0d4b-453e-ac45-84fea7a86bd2-kube-api-access-5hswd\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:37:34.692877 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.692844    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerID="4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688" exitCode=0
Jul 28 01:37:34.693300 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.692895    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"
Jul 28 01:37:34.693300 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.692931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerDied","Data":"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"}
Jul 28 01:37:34.693300 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.692993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h" event={"ID":"5bdec550-0d4b-453e-ac45-84fea7a86bd2","Type":"ContainerDied","Data":"a5d63978d05867591edb9fe516e23047e1660f471d0e11d429a117a8d96987a7"}
Jul 28 01:37:34.693300 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.693020    2581 scope.go:117] "RemoveContainer" containerID="63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"
Jul 28 01:37:34.701816 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.701781    2581 scope.go:117] "RemoveContainer" containerID="4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"
Jul 28 01:37:34.711551 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.711529    2581 scope.go:117] "RemoveContainer" containerID="63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"
Jul 28 01:37:34.712069 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:37:34.711888    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493\": container with ID starting with 63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493 not found: ID does not exist" containerID="63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"
Jul 28 01:37:34.712069 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.711926    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493"} err="failed to get container status \"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493\": rpc error: code = NotFound desc = could not find container \"63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493\": container with ID starting with 63df16fb071989c5d886f5b6d7aa6b22f0ffd4223dd5ec3d97386df5bd100493 not found: ID does not exist"
Jul 28 01:37:34.712069 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.711952    2581 scope.go:117] "RemoveContainer" containerID="4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"
Jul 28 01:37:34.713404 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:37:34.713370    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688\": container with ID starting with 4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688 not found: ID does not exist" containerID="4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"
Jul 28 01:37:34.713510 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.713414    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688"} err="failed to get container status \"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688\": rpc error: code = NotFound desc = could not find container \"4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688\": container with ID starting with 4666d0b2b7a5b013028a62759d0ea125f27a1ea8c0c01d46487a0e93c0b21688 not found: ID does not exist"
Jul 28 01:37:34.714295 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.714277    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:37:34.718224 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:34.718198    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7854-predictor-644bcbbbf7-7k64h"]
Jul 28 01:37:35.647684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:35.647653    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" path="/var/lib/kubelet/pods/5bdec550-0d4b-453e-ac45-84fea7a86bd2/volumes"
Jul 28 01:37:36.423545 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:36.423512    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:37:38.692572 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:38.692540    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:37:38.693040 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:38.693013    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:37:39.338455 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:39.338413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:37:40.345003 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:40.344975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:37:44.914920 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:44.914819    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:44.972879 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:44.972846    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:37:48.693999 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:48.693949    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:37:58.693907 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:37:58.693861    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:38:00.327199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:00.327160    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:00.557357 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:00.557317    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:08.693542 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:08.693496    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:38:10.424993 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.424936    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.511630 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.511599    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:38:10.511924 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.511899    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container" containerID="cri-o://0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc" gracePeriod=30
Jul 28 01:38:10.512010 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.511901    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kube-rbac-proxy" containerID="cri-o://1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6" gracePeriod=30
Jul 28 01:38:10.549972 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.547049    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.568787 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.568751    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:38:10.569124 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.569076    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container" containerID="cri-o://beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d" gracePeriod=30
Jul 28 01:38:10.569232 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.569137    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kube-rbac-proxy" containerID="cri-o://c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5" gracePeriod=30
Jul 28 01:38:10.598889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.598852    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:38:10.599208 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599196    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kube-rbac-proxy"
Jul 28 01:38:10.599291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599210    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kube-rbac-proxy"
Jul 28 01:38:10.599291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599226    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container"
Jul 28 01:38:10.599291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599231    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container"
Jul 28 01:38:10.599385 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599301    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kserve-container"
Jul 28 01:38:10.599385 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.599313    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdec550-0d4b-453e-ac45-84fea7a86bd2" containerName="kube-rbac-proxy"
Jul 28 01:38:10.602741 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.602721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.608467 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.608408    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3d51c-predictor-serving-cert\""
Jul 28 01:38:10.608467 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.608429    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-3d51c-kube-rbac-proxy-sar-config\""
Jul 28 01:38:10.618040 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.618008    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:38:10.663088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.663052    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:38:10.663309 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.663288    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container" containerID="cri-o://9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11" gracePeriod=30
Jul 28 01:38:10.663378 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.663318    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kube-rbac-proxy" containerID="cri-o://f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42" gracePeriod=30
Jul 28 01:38:10.673270 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.673217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.673464 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.673361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.673464 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.673422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkxfc\" (UniqueName: \"kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.683423 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.683380    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.774580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.774540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.774762 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.774592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkxfc\" (UniqueName: \"kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.774762 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.774666    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.774904 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:10.774784    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-serving-cert: secret "success-200-isvc-3d51c-predictor-serving-cert" not found
Jul 28 01:38:10.774904 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:10.774865    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls podName:9e66766a-1d6b-4d63-9ea2-6bfd489b8086 nodeName:}" failed. No retries permitted until 2026-07-28 01:38:11.27484771 +0000 UTC m=+740.180417104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls") pod "success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" (UID: "9e66766a-1d6b-4d63-9ea2-6bfd489b8086") : secret "success-200-isvc-3d51c-predictor-serving-cert" not found
Jul 28 01:38:10.775281 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.775257    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.780382 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.780239    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:10.786862 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.786835    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkxfc\" (UniqueName: \"kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:10.849961 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.849921    2581 generic.go:356] "Generic (PLEG): container finished" podID="87fd680b-b995-49c6-ad0d-62c45a833494" containerID="1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6" exitCode=2
Jul 28 01:38:10.850132 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.849993    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerDied","Data":"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"}
Jul 28 01:38:10.852242 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.852218    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerID="c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5" exitCode=2
Jul 28 01:38:10.852363 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.852286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerDied","Data":"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"}
Jul 28 01:38:10.854204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.854181    2581 generic.go:356] "Generic (PLEG): container finished" podID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerID="f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42" exitCode=2
Jul 28 01:38:10.854301 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.854241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerDied","Data":"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"}
Jul 28 01:38:10.975126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:10.975044    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:11.256629 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.256599    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:11.278698 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.278654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:11.281187 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.281157    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") pod \"success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") " pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:11.418868 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.418822    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 01:38:11.512903 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.512791    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:11.651251 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.651219    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:38:11.653446 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:38:11.653416    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e66766a_1d6b_4d63_9ea2_6bfd489b8086.slice/crio-b9c8704375478080a06cec345411a5f9dcbf53eca55fec6ee3cfbb1c12a8e2bf WatchSource:0}: Error finding container b9c8704375478080a06cec345411a5f9dcbf53eca55fec6ee3cfbb1c12a8e2bf: Status 404 returned error can't find the container with id b9c8704375478080a06cec345411a5f9dcbf53eca55fec6ee3cfbb1c12a8e2bf
Jul 28 01:38:11.860964 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.860932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerStarted","Data":"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"}
Jul 28 01:38:11.861135 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.860989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerStarted","Data":"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"}
Jul 28 01:38:11.861135 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.861011    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerStarted","Data":"b9c8704375478080a06cec345411a5f9dcbf53eca55fec6ee3cfbb1c12a8e2bf"}
Jul 28 01:38:11.861135 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.861067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:11.881920 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:11.881856    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podStartSLOduration=1.881819142 podStartE2EDuration="1.881819142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:38:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:38:11.88045605 +0000 UTC m=+740.786025449" watchObservedRunningTime="2026-07-28 01:38:11.881819142 +0000 UTC m=+740.787388540"
Jul 28 01:38:12.865033 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:12.865001    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:12.866460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:12.866426    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:13.870560 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:13.870513    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:14.332042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:14.331994    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Jul 28 01:38:15.014507 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.014480    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:38:15.115241 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115141    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bd5wr\" (UniqueName: \"kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr\") pod \"d23e5073-7566-4811-9ffc-96d78ea1d092\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") "
Jul 28 01:38:15.115241 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115200    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config\") pod \"d23e5073-7566-4811-9ffc-96d78ea1d092\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") "
Jul 28 01:38:15.115241 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115245    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location\") pod \"d23e5073-7566-4811-9ffc-96d78ea1d092\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") "
Jul 28 01:38:15.115521 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115311    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert\") pod \"d23e5073-7566-4811-9ffc-96d78ea1d092\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") "
Jul 28 01:38:15.115521 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115346    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls\") pod \"d23e5073-7566-4811-9ffc-96d78ea1d092\" (UID: \"d23e5073-7566-4811-9ffc-96d78ea1d092\") "
Jul 28 01:38:15.115658 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d23e5073-7566-4811-9ffc-96d78ea1d092" (UID: "d23e5073-7566-4811-9ffc-96d78ea1d092"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:15.115737 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115629    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d23e5073-7566-4811-9ffc-96d78ea1d092" (UID: "d23e5073-7566-4811-9ffc-96d78ea1d092"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:15.115737 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.115668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d23e5073-7566-4811-9ffc-96d78ea1d092" (UID: "d23e5073-7566-4811-9ffc-96d78ea1d092"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:15.117325 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.117300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr" (OuterVolumeSpecName: "kube-api-access-bd5wr") pod "d23e5073-7566-4811-9ffc-96d78ea1d092" (UID: "d23e5073-7566-4811-9ffc-96d78ea1d092"). InnerVolumeSpecName "kube-api-access-bd5wr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:15.117415 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.117383    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d23e5073-7566-4811-9ffc-96d78ea1d092" (UID: "d23e5073-7566-4811-9ffc-96d78ea1d092"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:15.216541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.216501    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5wr\" (UniqueName: \"kubernetes.io/projected/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-api-access-bd5wr\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:15.216541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.216529    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:15.216541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.216538    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d23e5073-7566-4811-9ffc-96d78ea1d092-kserve-provision-location\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:15.216541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.216547    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d23e5073-7566-4811-9ffc-96d78ea1d092-cabundle-cert\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:15.216826 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.216559    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d23e5073-7566-4811-9ffc-96d78ea1d092-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:15.336986 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.336932    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 01:38:15.880840 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.880776    2581 generic.go:356] "Generic (PLEG): container finished" podID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerID="9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11" exitCode=0
Jul 28 01:38:15.881041 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.880865    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"
Jul 28 01:38:15.881041 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.880860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerDied","Data":"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"}
Jul 28 01:38:15.881041 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.880999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6" event={"ID":"d23e5073-7566-4811-9ffc-96d78ea1d092","Type":"ContainerDied","Data":"12d93e703c0dae34a3953396d1bcc9eb982146e979e8c727b9cfd912ab3028d5"}
Jul 28 01:38:15.881041 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.881027    2581 scope.go:117] "RemoveContainer" containerID="f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"
Jul 28 01:38:15.889541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.889522    2581 scope.go:117] "RemoveContainer" containerID="9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"
Jul 28 01:38:15.897609 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.897586    2581 scope.go:117] "RemoveContainer" containerID="6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"
Jul 28 01:38:15.900996 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.900969    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:38:15.906640 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.906618    2581 scope.go:117] "RemoveContainer" containerID="f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"
Jul 28 01:38:15.906755 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.906734    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-predictor-555fc69d56-b2mj6"]
Jul 28 01:38:15.906973 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:15.906955    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42\": container with ID starting with f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42 not found: ID does not exist" containerID="f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"
Jul 28 01:38:15.907025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.906983    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42"} err="failed to get container status \"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42\": rpc error: code = NotFound desc = could not find container \"f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42\": container with ID starting with f84aa498ae97c31d4a001c0ceb34af7e10ee6428ffdae909ba19a04b4d0a3d42 not found: ID does not exist"
Jul 28 01:38:15.907025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.907001    2581 scope.go:117] "RemoveContainer" containerID="9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"
Jul 28 01:38:15.907253 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:15.907237    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11\": container with ID starting with 9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11 not found: ID does not exist" containerID="9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"
Jul 28 01:38:15.907319 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.907262    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11"} err="failed to get container status \"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11\": rpc error: code = NotFound desc = could not find container \"9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11\": container with ID starting with 9d5e4b8d723e0d369e55f8b7648e10f976b0ef3bbb7113dc22c9195e362e1d11 not found: ID does not exist"
Jul 28 01:38:15.907319 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.907287    2581 scope.go:117] "RemoveContainer" containerID="6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"
Jul 28 01:38:15.907507 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:15.907492    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a\": container with ID starting with 6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a not found: ID does not exist" containerID="6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"
Jul 28 01:38:15.907545 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:15.907511    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a"} err="failed to get container status \"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a\": rpc error: code = NotFound desc = could not find container \"6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a\": container with ID starting with 6bfcb89de818d93246070b3bce1a920eb2e208191087a54b042e07c24f3b148a not found: ID does not exist"
Jul 28 01:38:16.335381 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.335348    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:38:16.428192 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config\") pod \"ffa80abb-6239-4e15-b294-cfde80f6becf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") "
Jul 28 01:38:16.428399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428211    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert\") pod \"ffa80abb-6239-4e15-b294-cfde80f6becf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") "
Jul 28 01:38:16.428399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f8hmh\" (UniqueName: \"kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh\") pod \"ffa80abb-6239-4e15-b294-cfde80f6becf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") "
Jul 28 01:38:16.428399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428282    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") pod \"ffa80abb-6239-4e15-b294-cfde80f6becf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") "
Jul 28 01:38:16.428399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428319    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location\") pod \"ffa80abb-6239-4e15-b294-cfde80f6becf\" (UID: \"ffa80abb-6239-4e15-b294-cfde80f6becf\") "
Jul 28 01:38:16.428682 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428649    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffa80abb-6239-4e15-b294-cfde80f6becf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:16.428790 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428674    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ffa80abb-6239-4e15-b294-cfde80f6becf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:16.428790 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.428699    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ffa80abb-6239-4e15-b294-cfde80f6becf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:16.430866 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.430836    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffa80abb-6239-4e15-b294-cfde80f6becf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:16.430967 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.430904    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh" (OuterVolumeSpecName: "kube-api-access-f8hmh") pod "ffa80abb-6239-4e15-b294-cfde80f6becf" (UID: "ffa80abb-6239-4e15-b294-cfde80f6becf"). InnerVolumeSpecName "kube-api-access-f8hmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:16.452890 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.452864    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:38:16.529254 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529214    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert\") pod \"87fd680b-b995-49c6-ad0d-62c45a833494\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") "
Jul 28 01:38:16.529456 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529294    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config\") pod \"87fd680b-b995-49c6-ad0d-62c45a833494\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") "
Jul 28 01:38:16.529456 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529375    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls\") pod \"87fd680b-b995-49c6-ad0d-62c45a833494\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") "
Jul 28 01:38:16.529456 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529432    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location\") pod \"87fd680b-b995-49c6-ad0d-62c45a833494\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") "
Jul 28 01:38:16.529611 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529457    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s7zbs\" (UniqueName: \"kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs\") pod \"87fd680b-b995-49c6-ad0d-62c45a833494\" (UID: \"87fd680b-b995-49c6-ad0d-62c45a833494\") "
Jul 28 01:38:16.529707 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529679    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "87fd680b-b995-49c6-ad0d-62c45a833494" (UID: "87fd680b-b995-49c6-ad0d-62c45a833494"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:16.529752 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529693    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-cabundle-cert\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.529752 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529739    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8hmh\" (UniqueName: \"kubernetes.io/projected/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-api-access-f8hmh\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.529864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529752    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffa80abb-6239-4e15-b294-cfde80f6becf-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.529864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529713    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "87fd680b-b995-49c6-ad0d-62c45a833494" (UID: "87fd680b-b995-49c6-ad0d-62c45a833494"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:38:16.529864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529761    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ffa80abb-6239-4e15-b294-cfde80f6becf-kserve-provision-location\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.529864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529741    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "87fd680b-b995-49c6-ad0d-62c45a833494" (UID: "87fd680b-b995-49c6-ad0d-62c45a833494"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 01:38:16.529864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.529771    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffa80abb-6239-4e15-b294-cfde80f6becf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.531566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.531541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "87fd680b-b995-49c6-ad0d-62c45a833494" (UID: "87fd680b-b995-49c6-ad0d-62c45a833494"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:38:16.531683 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.531576    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs" (OuterVolumeSpecName: "kube-api-access-s7zbs") pod "87fd680b-b995-49c6-ad0d-62c45a833494" (UID: "87fd680b-b995-49c6-ad0d-62c45a833494"). InnerVolumeSpecName "kube-api-access-s7zbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:38:16.630480 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.630441    2581 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/87fd680b-b995-49c6-ad0d-62c45a833494-kserve-provision-location\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.630480 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.630473    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7zbs\" (UniqueName: \"kubernetes.io/projected/87fd680b-b995-49c6-ad0d-62c45a833494-kube-api-access-s7zbs\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.630480 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.630487    2581 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-cabundle-cert\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.630726 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.630498    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/87fd680b-b995-49c6-ad0d-62c45a833494-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.630726 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.630509    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/87fd680b-b995-49c6-ad0d-62c45a833494-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:38:16.887985 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.887946    2581 generic.go:356] "Generic (PLEG): container finished" podID="87fd680b-b995-49c6-ad0d-62c45a833494" containerID="0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc" exitCode=0
Jul 28 01:38:16.888195 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.888021    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"
Jul 28 01:38:16.888195 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.888040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerDied","Data":"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"}
Jul 28 01:38:16.888195 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.888117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k" event={"ID":"87fd680b-b995-49c6-ad0d-62c45a833494","Type":"ContainerDied","Data":"f935591f3df571a3b8d22a41970d2f21527ab7177c94ee8a87ca6824da1c4e7d"}
Jul 28 01:38:16.888195 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.888155    2581 scope.go:117] "RemoveContainer" containerID="1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"
Jul 28 01:38:16.890074 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.890050    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerID="beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d" exitCode=0
Jul 28 01:38:16.890174 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.890099    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"
Jul 28 01:38:16.890174 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.890140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerDied","Data":"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"}
Jul 28 01:38:16.890278 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.890206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf" event={"ID":"ffa80abb-6239-4e15-b294-cfde80f6becf","Type":"ContainerDied","Data":"0fd5c40636713e9c66a700f6a1dbf17d699a837dff8515a2e9d1141cc63c91b6"}
Jul 28 01:38:16.898210 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.898192    2581 scope.go:117] "RemoveContainer" containerID="0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"
Jul 28 01:38:16.906298 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.906275    2581 scope.go:117] "RemoveContainer" containerID="39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"
Jul 28 01:38:16.913759 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.913727    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:38:16.922772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.916398    2581 scope.go:117] "RemoveContainer" containerID="1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"
Jul 28 01:38:16.922772 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.916822    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6\": container with ID starting with 1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6 not found: ID does not exist" containerID="1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"
Jul 28 01:38:16.922772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.916864    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6"} err="failed to get container status \"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6\": rpc error: code = NotFound desc = could not find container \"1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6\": container with ID starting with 1c3121989c7dfd5ae76fbc0a7e1eddd060e28a80eb4af98a433c8e413f189ea6 not found: ID does not exist"
Jul 28 01:38:16.922772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.916891    2581 scope.go:117] "RemoveContainer" containerID="0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"
Jul 28 01:38:16.922772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.918660    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-2-predictor-7d44765f95-zc72k"]
Jul 28 01:38:16.923088 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.922932    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc\": container with ID starting with 0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc not found: ID does not exist" containerID="0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"
Jul 28 01:38:16.923138 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.923120    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc"} err="failed to get container status \"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc\": rpc error: code = NotFound desc = could not find container \"0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc\": container with ID starting with 0c7c96926d56d3bf36bdcdaa570a3ff5c548121be68eb63854ef609c38a2c7bc not found: ID does not exist"
Jul 28 01:38:16.923176 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.923142    2581 scope.go:117] "RemoveContainer" containerID="39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"
Jul 28 01:38:16.923463 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.923445    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d\": container with ID starting with 39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d not found: ID does not exist" containerID="39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"
Jul 28 01:38:16.923532 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.923468    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d"} err="failed to get container status \"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d\": rpc error: code = NotFound desc = could not find container \"39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d\": container with ID starting with 39d4fedcacca30dea70fee52c278364cfc948fde2c2a7aec3deca4947015955d not found: ID does not exist"
Jul 28 01:38:16.923532 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.923484    2581 scope.go:117] "RemoveContainer" containerID="c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"
Jul 28 01:38:16.928126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.928102    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:38:16.931896 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.931866    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-1-predictor-6d55f5d589-jvccf"]
Jul 28 01:38:16.932844 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.932792    2581 scope.go:117] "RemoveContainer" containerID="beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"
Jul 28 01:38:16.941958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.941929    2581 scope.go:117] "RemoveContainer" containerID="e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"
Jul 28 01:38:16.949952 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.949923    2581 scope.go:117] "RemoveContainer" containerID="c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"
Jul 28 01:38:16.950274 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.950256    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5\": container with ID starting with c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5 not found: ID does not exist" containerID="c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"
Jul 28 01:38:16.950318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.950284    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5"} err="failed to get container status \"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5\": rpc error: code = NotFound desc = could not find container \"c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5\": container with ID starting with c8647594246108942eca4b72ea611473d75fb6dcb4e81423da548448b07f94d5 not found: ID does not exist"
Jul 28 01:38:16.950318 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.950304    2581 scope.go:117] "RemoveContainer" containerID="beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"
Jul 28 01:38:16.950538 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.950519    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d\": container with ID starting with beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d not found: ID does not exist" containerID="beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"
Jul 28 01:38:16.950581 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.950545    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d"} err="failed to get container status \"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d\": rpc error: code = NotFound desc = could not find container \"beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d\": container with ID starting with beabcf11cebf87c2c398ad725d6d9b4f468aa8511a1ce4f493b8c2142aec9a4d not found: ID does not exist"
Jul 28 01:38:16.950581 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.950561    2581 scope.go:117] "RemoveContainer" containerID="e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"
Jul 28 01:38:16.950785 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:38:16.950771    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b\": container with ID starting with e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b not found: ID does not exist" containerID="e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"
Jul 28 01:38:16.950906 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:16.950789    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b"} err="failed to get container status \"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b\": rpc error: code = NotFound desc = could not find container \"e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b\": container with ID starting with e31432a07787b80a465aa8ad24bd1f859eb32a4c50b445beacbefe4449c5448b not found: ID does not exist"
Jul 28 01:38:17.647458 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:17.647427    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" path="/var/lib/kubelet/pods/87fd680b-b995-49c6-ad0d-62c45a833494/volumes"
Jul 28 01:38:17.647932 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:17.647919    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" path="/var/lib/kubelet/pods/d23e5073-7566-4811-9ffc-96d78ea1d092/volumes"
Jul 28 01:38:17.648646 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:17.648632    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" path="/var/lib/kubelet/pods/ffa80abb-6239-4e15-b294-cfde80f6becf/volumes"
Jul 28 01:38:18.693671 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:18.693631    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:38:18.878459 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:18.878425    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:38:18.879029 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:18.879006    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:28.694629 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:28.694597    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:38:28.879526 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:28.879484    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:38.880025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:38.879980    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:40.808703 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:40.808662    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:41.034963 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:41.034913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:43.077808 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:43.077752    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:38:48.879775 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:48.879732    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 01:38:58.879951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:38:58.879920    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:39:10.617975 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.617931    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618268    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618280    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618293    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618299    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618305    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618311    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618317    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618323    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618329    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618334    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618344    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618349    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618359    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618364    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="storage-initializer"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618372    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618377    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618386    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container"
Jul 28 01:39:10.618403 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618391    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618439    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kserve-container"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618446    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kserve-container"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618452    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa80abb-6239-4e15-b294-cfde80f6becf" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618466    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d23e5073-7566-4811-9ffc-96d78ea1d092" containerName="kube-rbac-proxy"
Jul 28 01:39:10.618958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.618471    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87fd680b-b995-49c6-ad0d-62c45a833494" containerName="kserve-container"
Jul 28 01:39:10.621925 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.621896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.622098 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.621968    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:10.624871 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.624846    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3d51c-kube-rbac-proxy-sar-config\""
Jul 28 01:39:10.624992 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.624846    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-3d51c-serving-cert\""
Jul 28 01:39:10.631392 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.631364    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:39:10.788321 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.788281    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.788646 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.788605    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.870642 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.870548    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:39:10.889873 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.889838    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.890042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.889896    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.890509 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.890485    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.892366 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.892343    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls\") pod \"sequence-graph-3d51c-7c5b587fbb-qmdr7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") " pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:10.932970 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:10.932930    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:11.066304 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:11.066279    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:39:11.068985 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:39:11.068954    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05a3502d_0259_43ca_8fee_9e12baeb59f7.slice/crio-5877cb34ac903af26cb1d7c95dc662565062bab88171a2bd0acb9dc3c56ff399 WatchSource:0}: Error finding container 5877cb34ac903af26cb1d7c95dc662565062bab88171a2bd0acb9dc3c56ff399: Status 404 returned error can't find the container with id 5877cb34ac903af26cb1d7c95dc662565062bab88171a2bd0acb9dc3c56ff399
Jul 28 01:39:11.070868 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:11.070847    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 01:39:11.118450 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:11.118413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" event={"ID":"05a3502d-0259-43ca-8fee-9e12baeb59f7","Type":"ContainerStarted","Data":"5877cb34ac903af26cb1d7c95dc662565062bab88171a2bd0acb9dc3c56ff399"}
Jul 28 01:39:13.128517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:13.128477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" event={"ID":"05a3502d-0259-43ca-8fee-9e12baeb59f7","Type":"ContainerStarted","Data":"6bd45c99be9a1fbbceb1807c8102162869056a73ae6c223fd71802e054454d05"}
Jul 28 01:39:13.128944 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:13.128652    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:39:13.145977 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:13.145868    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podStartSLOduration=1.328612903 podStartE2EDuration="3.145840888s" totalImagesPullingTime="1.817227985s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:39:13.144314122 +0000 UTC m=+802.049883519" watchObservedRunningTime="2026-07-28 01:39:13.145840888 +0000 UTC m=+802.051410315"
Jul 28 01:39:19.138439 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:39:19.138408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:40:51.596602 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:40:51.596522    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:40:51.597417 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:40:51.597400    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:45:51.623182 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:45:51.623154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:45:51.624539 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:45:51.624520    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:46:08.750949 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:08.750912    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.410443 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.410366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.537743 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.537707    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:46:55.538024 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.537972    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" containerID="cri-o://ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e" gracePeriod=30
Jul 28 01:46:55.538146 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.538002    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kube-rbac-proxy" containerID="cri-o://585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c" gracePeriod=30
Jul 28 01:46:55.559679 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.559646    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.604562 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.604523    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:46:55.608120 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.608095    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.611030 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.611003    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-02c81-predictor-serving-cert\""
Jul 28 01:46:55.611167 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.611002    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-02c81-kube-rbac-proxy-sar-config\""
Jul 28 01:46:55.622753 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.622725    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:46:55.646620 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.646566    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.686751 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.686662    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kkb9\" (UniqueName: \"kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.686925 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.686776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.686925 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.686829    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.727876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.727840    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.787692 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.787657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.787938 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.787701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kkb9\" (UniqueName: \"kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.787938 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.787791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.788523 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.788500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.790221 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.790194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.795705 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.795681    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kkb9\" (UniqueName: \"kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9\") pod \"success-200-isvc-02c81-predictor-5b98d76d87-zqtqz\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") " pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.827250 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.827193    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.922124 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.922086    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:55.927096 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.927066    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:46:55.969364 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.969274    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerID="585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c" exitCode=2
Jul 28 01:46:55.969364 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:55.969341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerDied","Data":"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"}
Jul 28 01:46:56.056221 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.056192    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:46:56.058888 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:46:56.058865    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c45a3e_cc55_4400_99ea_f97663c7f77c.slice/crio-f5e67033a5457ca0694427fad4d5048ab9a12b9c545c17a2b60f573dd9b2cef7 WatchSource:0}: Error finding container f5e67033a5457ca0694427fad4d5048ab9a12b9c545c17a2b60f573dd9b2cef7: Status 404 returned error can't find the container with id f5e67033a5457ca0694427fad4d5048ab9a12b9c545c17a2b60f573dd9b2cef7
Jul 28 01:46:56.974958 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.974922    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerStarted","Data":"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"}
Jul 28 01:46:56.975397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.975007    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:56.975397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.975028    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerStarted","Data":"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"}
Jul 28 01:46:56.975397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.975049    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:46:56.975397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.975066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerStarted","Data":"f5e67033a5457ca0694427fad4d5048ab9a12b9c545c17a2b60f573dd9b2cef7"}
Jul 28 01:46:56.976324 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.976302    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:46:56.993016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:56.992969    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podStartSLOduration=1.9929531489999999 podStartE2EDuration="1.992953149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:46:56.991662809 +0000 UTC m=+1265.897232207" watchObservedRunningTime="2026-07-28 01:46:56.992953149 +0000 UTC m=+1265.898522547"
Jul 28 01:46:57.979160 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:57.979114    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:46:58.688018 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.687973    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.25:8643/healthz\": dial tcp 10.134.0.25:8643: connect: connection refused"
Jul 28 01:46:58.693126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.693086    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 01:46:58.793691 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.793663    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:46:58.917531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.917429    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") pod \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") "
Jul 28 01:46:58.917531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.917490    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9gg2l\" (UniqueName: \"kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l\") pod \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") "
Jul 28 01:46:58.917531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.917522    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config\") pod \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\" (UID: \"ffacdfd9-fb25-4a70-b282-36051a7dcafc\") "
Jul 28 01:46:58.917999 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.917973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ffacdfd9-fb25-4a70-b282-36051a7dcafc" (UID: "ffacdfd9-fb25-4a70-b282-36051a7dcafc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:46:58.919635 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.919610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ffacdfd9-fb25-4a70-b282-36051a7dcafc" (UID: "ffacdfd9-fb25-4a70-b282-36051a7dcafc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:46:58.919635 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.919620    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l" (OuterVolumeSpecName: "kube-api-access-9gg2l") pod "ffacdfd9-fb25-4a70-b282-36051a7dcafc" (UID: "ffacdfd9-fb25-4a70-b282-36051a7dcafc"). InnerVolumeSpecName "kube-api-access-9gg2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:46:58.984009 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.983972    2581 generic.go:356] "Generic (PLEG): container finished" podID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerID="ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e" exitCode=0
Jul 28 01:46:58.984414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.984037    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"
Jul 28 01:46:58.984414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.984055    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerDied","Data":"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"}
Jul 28 01:46:58.984414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.984116    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f" event={"ID":"ffacdfd9-fb25-4a70-b282-36051a7dcafc","Type":"ContainerDied","Data":"8151cc1c5c4aba59beb8113c5c4e4b36da484ce998acd710a86825d120184d8d"}
Jul 28 01:46:58.984414 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.984151    2581 scope.go:117] "RemoveContainer" containerID="585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"
Jul 28 01:46:58.992921 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:58.992904    2581 scope.go:117] "RemoveContainer" containerID="ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"
Jul 28 01:46:59.000821 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.000779    2581 scope.go:117] "RemoveContainer" containerID="585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"
Jul 28 01:46:59.001123 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:46:59.001101    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c\": container with ID starting with 585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c not found: ID does not exist" containerID="585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"
Jul 28 01:46:59.001209 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.001135    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c"} err="failed to get container status \"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c\": rpc error: code = NotFound desc = could not find container \"585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c\": container with ID starting with 585b3da329cfe18783f1db63ef3a72e2feb3dd91dbb4a63a61122551d5b57e1c not found: ID does not exist"
Jul 28 01:46:59.001209 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.001159    2581 scope.go:117] "RemoveContainer" containerID="ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"
Jul 28 01:46:59.001532 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:46:59.001507    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e\": container with ID starting with ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e not found: ID does not exist" containerID="ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"
Jul 28 01:46:59.001626 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.001541    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e"} err="failed to get container status \"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e\": rpc error: code = NotFound desc = could not find container \"ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e\": container with ID starting with ca946e7e12a842cfa6e9351b903fab2973fe9347a231e1f23b6734e9c730180e not found: ID does not exist"
Jul 28 01:46:59.007707 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.007681    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:46:59.012580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.012556    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-33843-predictor-78f5cf5cd9-vzq7f"]
Jul 28 01:46:59.018362 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.018340    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.018362 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.018363    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ffacdfd9-fb25-4a70-b282-36051a7dcafc-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.018491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.018375    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gg2l\" (UniqueName: \"kubernetes.io/projected/ffacdfd9-fb25-4a70-b282-36051a7dcafc-kube-api-access-9gg2l\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:46:59.647724 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:46:59.647688    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" path="/var/lib/kubelet/pods/ffacdfd9-fb25-4a70-b282-36051a7dcafc/volumes"
Jul 28 01:47:02.983810 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:02.983768    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:47:02.984442 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:02.984415    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:47:12.984521 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:12.984478    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:47:22.985077 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:22.985029    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:47:25.146575 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.146534    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:47:25.147014 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.146737    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" containerID="cri-o://6bd45c99be9a1fbbceb1807c8102162869056a73ae6c223fd71802e054454d05" gracePeriod=30
Jul 28 01:47:25.174500 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.174461    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:25.252546 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.252511    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:47:25.252853 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.252826    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" containerID="cri-o://5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d" gracePeriod=30
Jul 28 01:47:25.252946 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.252857    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kube-rbac-proxy" containerID="cri-o://be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b" gracePeriod=30
Jul 28 01:47:25.289948 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.289905    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:25.352133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352100    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:47:25.352471 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352441    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kube-rbac-proxy"
Jul 28 01:47:25.352516 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352475    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kube-rbac-proxy"
Jul 28 01:47:25.352516 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352501    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container"
Jul 28 01:47:25.352516 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container"
Jul 28 01:47:25.352611 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352555    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kube-rbac-proxy"
Jul 28 01:47:25.352611 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.352565    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffacdfd9-fb25-4a70-b282-36051a7dcafc" containerName="kserve-container"
Jul 28 01:47:25.355665 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.355647    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.357886 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.357863    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2a946-predictor-serving-cert\""
Jul 28 01:47:25.358056 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.357904    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-2a946-kube-rbac-proxy-sar-config\""
Jul 28 01:47:25.361677 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.361636    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:25.365420 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.365389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:47:25.435162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.435057    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:25.536871 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.536832    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:25.547780 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.547748    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.547974 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.547873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.547974 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.547914    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vx54g\" (UniqueName: \"kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.649361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.649327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.649574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.649486    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.649574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.649526    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vx54g\" (UniqueName: \"kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.650117 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.650099    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.651788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.651769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.657133 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.657103    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vx54g\" (UniqueName: \"kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g\") pod \"success-200-isvc-2a946-predictor-86c64754c9-psq5l\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") " pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.667884 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.667846    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:25.802792 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:25.802767    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:47:25.805081 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:47:25.805056    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b8eaec4_835a_43cf_b192_aa4bbe5ef0e7.slice/crio-21437a35e190c8e17d4c49239029de92fd01c3e0e526db2d440d1253c34cf128 WatchSource:0}: Error finding container 21437a35e190c8e17d4c49239029de92fd01c3e0e526db2d440d1253c34cf128: Status 404 returned error can't find the container with id 21437a35e190c8e17d4c49239029de92fd01c3e0e526db2d440d1253c34cf128
Jul 28 01:47:26.027278 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.027250    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:47:26.091501 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.091468    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerID="be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b" exitCode=2
Jul 28 01:47:26.091671 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.091539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerDied","Data":"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"}
Jul 28 01:47:26.093296 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.093273    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerStarted","Data":"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"}
Jul 28 01:47:26.093406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.093318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerStarted","Data":"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"}
Jul 28 01:47:26.093406 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.093344    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerStarted","Data":"21437a35e190c8e17d4c49239029de92fd01c3e0e526db2d440d1253c34cf128"}
Jul 28 01:47:26.093494 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.093478    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:26.093626 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.093608    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:26.095021 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.094996    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:47:26.112548 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:26.112497    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podStartSLOduration=1.112481004 podStartE2EDuration="1.112481004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:47:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:47:26.111454871 +0000 UTC m=+1295.017024281" watchObservedRunningTime="2026-07-28 01:47:26.112481004 +0000 UTC m=+1295.018050402"
Jul 28 01:47:27.097683 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:27.097641    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:47:28.904230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:28.904195    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:47:29.078305 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.078270    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") pod \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") "
Jul 28 01:47:29.078517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.078383    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config\") pod \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") "
Jul 28 01:47:29.078517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.078417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gkxfc\" (UniqueName: \"kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc\") pod \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\" (UID: \"9e66766a-1d6b-4d63-9ea2-6bfd489b8086\") "
Jul 28 01:47:29.078849 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.078783    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9e66766a-1d6b-4d63-9ea2-6bfd489b8086" (UID: "9e66766a-1d6b-4d63-9ea2-6bfd489b8086"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:29.080462 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.080433    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9e66766a-1d6b-4d63-9ea2-6bfd489b8086" (UID: "9e66766a-1d6b-4d63-9ea2-6bfd489b8086"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:29.080560 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.080461    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc" (OuterVolumeSpecName: "kube-api-access-gkxfc") pod "9e66766a-1d6b-4d63-9ea2-6bfd489b8086" (UID: "9e66766a-1d6b-4d63-9ea2-6bfd489b8086"). InnerVolumeSpecName "kube-api-access-gkxfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:47:29.106622 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.106580    2581 generic.go:356] "Generic (PLEG): container finished" podID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerID="5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d" exitCode=0
Jul 28 01:47:29.106784 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.106638    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"
Jul 28 01:47:29.106784 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.106666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerDied","Data":"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"}
Jul 28 01:47:29.106784 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.106733    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" event={"ID":"9e66766a-1d6b-4d63-9ea2-6bfd489b8086","Type":"ContainerDied","Data":"b9c8704375478080a06cec345411a5f9dcbf53eca55fec6ee3cfbb1c12a8e2bf"}
Jul 28 01:47:29.106784 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.106763    2581 scope.go:117] "RemoveContainer" containerID="be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"
Jul 28 01:47:29.115831 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.115790    2581 scope.go:117] "RemoveContainer" containerID="5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"
Jul 28 01:47:29.123988 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.123966    2581 scope.go:117] "RemoveContainer" containerID="be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"
Jul 28 01:47:29.124285 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:47:29.124266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b\": container with ID starting with be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b not found: ID does not exist" containerID="be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"
Jul 28 01:47:29.124375 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.124296    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b"} err="failed to get container status \"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b\": rpc error: code = NotFound desc = could not find container \"be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b\": container with ID starting with be2469bf673c0b3337cab2d6b1c580ce9530a8baaa7d1a09303378a393b48e0b not found: ID does not exist"
Jul 28 01:47:29.124375 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.124316    2581 scope.go:117] "RemoveContainer" containerID="5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"
Jul 28 01:47:29.124621 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:47:29.124583    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d\": container with ID starting with 5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d not found: ID does not exist" containerID="5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"
Jul 28 01:47:29.124729 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.124624    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d"} err="failed to get container status \"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d\": rpc error: code = NotFound desc = could not find container \"5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d\": container with ID starting with 5ff6e3c68ac66213c09f2b14a442367db12875efa7c21b030126d7929458e06d not found: ID does not exist"
Jul 28 01:47:29.129132 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.129099    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:47:29.134361 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.134327    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np"]
Jul 28 01:47:29.137355 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.137322    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:29.178948 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.178914    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:47:29.178948 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.178945    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkxfc\" (UniqueName: \"kubernetes.io/projected/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-kube-api-access-gkxfc\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:47:29.178948 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.178957    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9e66766a-1d6b-4d63-9ea2-6bfd489b8086-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:47:29.647434 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.647401    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" path="/var/lib/kubelet/pods/9e66766a-1d6b-4d63-9ea2-6bfd489b8086/volumes"
Jul 28 01:47:29.872219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.872165    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 01:47:29.879113 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:29.879073    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-3d51c-predictor-7d7fc68cc5-fz2np" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: i/o timeout"
Jul 28 01:47:32.102768 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:32.102738    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:47:32.103284 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:32.103259    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:47:32.985382 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:32.985342    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:47:34.138299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:34.138251    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:39.137434 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:39.137397    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:39.137865 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:39.137513    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:47:42.103701 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:42.103662    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:47:42.984430 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:42.984389    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 01:47:44.138468 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:44.138426    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:49.137243 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:49.137200    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:52.103457 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:52.103418    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:47:52.985053 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:52.985023    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:47:54.137255 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:54.137212    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:47:55.215761 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.215727    2581 generic.go:356] "Generic (PLEG): container finished" podID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerID="6bd45c99be9a1fbbceb1807c8102162869056a73ae6c223fd71802e054454d05" exitCode=0
Jul 28 01:47:55.216164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.215775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" event={"ID":"05a3502d-0259-43ca-8fee-9e12baeb59f7","Type":"ContainerDied","Data":"6bd45c99be9a1fbbceb1807c8102162869056a73ae6c223fd71802e054454d05"}
Jul 28 01:47:55.797317 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.797287    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:47:55.902100 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.902059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle\") pod \"05a3502d-0259-43ca-8fee-9e12baeb59f7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") "
Jul 28 01:47:55.902301 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.902175    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls\") pod \"05a3502d-0259-43ca-8fee-9e12baeb59f7\" (UID: \"05a3502d-0259-43ca-8fee-9e12baeb59f7\") "
Jul 28 01:47:55.902509 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.902481    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "05a3502d-0259-43ca-8fee-9e12baeb59f7" (UID: "05a3502d-0259-43ca-8fee-9e12baeb59f7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:47:55.904200 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:55.904173    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "05a3502d-0259-43ca-8fee-9e12baeb59f7" (UID: "05a3502d-0259-43ca-8fee-9e12baeb59f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:47:56.003261 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.003232    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/05a3502d-0259-43ca-8fee-9e12baeb59f7-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:47:56.003261 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.003262    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/05a3502d-0259-43ca-8fee-9e12baeb59f7-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:47:56.220848 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.220815    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"
Jul 28 01:47:56.220848 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.220841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7" event={"ID":"05a3502d-0259-43ca-8fee-9e12baeb59f7","Type":"ContainerDied","Data":"5877cb34ac903af26cb1d7c95dc662565062bab88171a2bd0acb9dc3c56ff399"}
Jul 28 01:47:56.221330 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.220904    2581 scope.go:117] "RemoveContainer" containerID="6bd45c99be9a1fbbceb1807c8102162869056a73ae6c223fd71802e054454d05"
Jul 28 01:47:56.243039 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.243007    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:47:56.246858 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:56.246832    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-3d51c-7c5b587fbb-qmdr7"]
Jul 28 01:47:57.648350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:47:57.648316    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" path="/var/lib/kubelet/pods/05a3502d-0259-43ca-8fee-9e12baeb59f7/volumes"
Jul 28 01:48:02.103616 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:02.103574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:48:05.639334 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639254    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639599    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kube-rbac-proxy"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639611    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kube-rbac-proxy"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639642    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639648    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639672    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639722    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kserve-container"
Jul 28 01:48:05.639733 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="05a3502d-0259-43ca-8fee-9e12baeb59f7" containerName="sequence-graph-3d51c"
Jul 28 01:48:05.640009 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.639739    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e66766a-1d6b-4d63-9ea2-6bfd489b8086" containerName="kube-rbac-proxy"
Jul 28 01:48:05.643992 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.643960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.646350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.646319    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-02c81-serving-cert\""
Jul 28 01:48:05.646497 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.646325    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-02c81-kube-rbac-proxy-sar-config\""
Jul 28 01:48:05.654248 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.654220    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:05.654831 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.654786    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:05.782039 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.782000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.782264 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.782141    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.835150 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.835115    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:05.883382 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.883348    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.883481 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.883422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.884084 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.884061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.885921 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.885901    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls\") pod \"ensemble-graph-02c81-7c5f497f56-bxss9\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") " pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:05.955680 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:05.955581    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:06.134208 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:06.134174    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:06.138055 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:48:06.138025    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0503ca4_cb66_4d74_81a9_77e30998df69.slice/crio-54c62f0701af119f701473d132d77b24316111b0fae96ca410035eb004cb3e80 WatchSource:0}: Error finding container 54c62f0701af119f701473d132d77b24316111b0fae96ca410035eb004cb3e80: Status 404 returned error can't find the container with id 54c62f0701af119f701473d132d77b24316111b0fae96ca410035eb004cb3e80
Jul 28 01:48:06.264889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:06.264847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" event={"ID":"d0503ca4-cb66-4d74-81a9-77e30998df69","Type":"ContainerStarted","Data":"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"}
Jul 28 01:48:06.265103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:06.264911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" event={"ID":"d0503ca4-cb66-4d74-81a9-77e30998df69","Type":"ContainerStarted","Data":"54c62f0701af119f701473d132d77b24316111b0fae96ca410035eb004cb3e80"}
Jul 28 01:48:06.265103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:06.265021    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:06.281694 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:06.281635    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podStartSLOduration=1.281617368 podStartE2EDuration="1.281617368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:06.279752292 +0000 UTC m=+1335.185321700" watchObservedRunningTime="2026-07-28 01:48:06.281617368 +0000 UTC m=+1335.187186766"
Jul 28 01:48:12.103654 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:12.103609    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 01:48:12.275379 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:12.275344    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:15.707905 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.707824    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:15.708278 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.708022    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" containerID="cri-o://4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc" gracePeriod=30
Jul 28 01:48:15.721189 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.721134    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:15.824724 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.824690    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:48:15.824967 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.824944    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container" containerID="cri-o://441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a" gracePeriod=30
Jul 28 01:48:15.825042 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.824969    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kube-rbac-proxy" containerID="cri-o://a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269" gracePeriod=30
Jul 28 01:48:15.849690 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.849521    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:15.892121 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.892088    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:48:15.895670 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.895646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:15.898839 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.898791    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5a4a0-predictor-serving-cert\""
Jul 28 01:48:15.898839 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.898830    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-5a4a0-kube-rbac-proxy-sar-config\""
Jul 28 01:48:15.905099 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.905062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:48:15.925914 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.925877    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:15.973937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.973838    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:15.973937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.973895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntg6k\" (UniqueName: \"kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:15.974136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:15.973949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.005775 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.005729    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:16.075008 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.074964    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.075228 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.075017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntg6k\" (UniqueName: \"kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.075228 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:16.075115    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-serving-cert: secret "success-200-isvc-5a4a0-predictor-serving-cert" not found
Jul 28 01:48:16.075228 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.075150    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.075228 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:16.075178    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls podName:a1cacbd6-d19e-4f79-99af-ea1ce733efc7 nodeName:}" failed. No retries permitted until 2026-07-28 01:48:16.575160247 +0000 UTC m=+1345.480729628 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls") pod "success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" (UID: "a1cacbd6-d19e-4f79-99af-ea1ce733efc7") : secret "success-200-isvc-5a4a0-predictor-serving-cert" not found
Jul 28 01:48:16.075864 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.075840    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.087109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.087081    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntg6k\" (UniqueName: \"kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.270034 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.269996    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:16.310217 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.310185    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerID="a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269" exitCode=2
Jul 28 01:48:16.310388 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.310258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerDied","Data":"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"}
Jul 28 01:48:16.370888 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.370851    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:16.579849 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.579726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.582182 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.582156    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") pod \"success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") " pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.806528 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.806488    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:16.942469 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:16.942368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:48:16.945467 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:48:16.945438    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1cacbd6_d19e_4f79_99af_ea1ce733efc7.slice/crio-0bba37df57e18a456bd72294cd882153781b6fb05b39ced1892dbb74e02d3844 WatchSource:0}: Error finding container 0bba37df57e18a456bd72294cd882153781b6fb05b39ced1892dbb74e02d3844: Status 404 returned error can't find the container with id 0bba37df57e18a456bd72294cd882153781b6fb05b39ced1892dbb74e02d3844
Jul 28 01:48:17.274162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.274118    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:17.317356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.317323    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerStarted","Data":"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"}
Jul 28 01:48:17.317557 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.317385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerStarted","Data":"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"}
Jul 28 01:48:17.317557 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.317413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerStarted","Data":"0bba37df57e18a456bd72294cd882153781b6fb05b39ced1892dbb74e02d3844"}
Jul 28 01:48:17.317557 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.317439    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:17.335547 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.335483    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podStartSLOduration=2.335461888 podStartE2EDuration="2.335461888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:17.334025207 +0000 UTC m=+1346.239594606" watchObservedRunningTime="2026-07-28 01:48:17.335461888 +0000 UTC m=+1346.241031287"
Jul 28 01:48:17.979701 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:17.979659    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 01:48:18.321533 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:18.321500    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:18.322946 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:18.322920    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:19.326061 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.326012    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:19.483207 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.483181    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:48:19.606357 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.606321    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls\") pod \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") "
Jul 28 01:48:19.606556 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.606380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config\") pod \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") "
Jul 28 01:48:19.606556 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.606414    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kkb9\" (UniqueName: \"kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9\") pod \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\" (UID: \"e1c45a3e-cc55-4400-99ea-f97663c7f77c\") "
Jul 28 01:48:19.606788 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.606766    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e1c45a3e-cc55-4400-99ea-f97663c7f77c" (UID: "e1c45a3e-cc55-4400-99ea-f97663c7f77c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:19.608580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.608555    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9" (OuterVolumeSpecName: "kube-api-access-7kkb9") pod "e1c45a3e-cc55-4400-99ea-f97663c7f77c" (UID: "e1c45a3e-cc55-4400-99ea-f97663c7f77c"). InnerVolumeSpecName "kube-api-access-7kkb9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:19.608580 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.608567    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e1c45a3e-cc55-4400-99ea-f97663c7f77c" (UID: "e1c45a3e-cc55-4400-99ea-f97663c7f77c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:19.707782 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.707744    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kkb9\" (UniqueName: \"kubernetes.io/projected/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-api-access-7kkb9\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:19.707782 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.707778    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e1c45a3e-cc55-4400-99ea-f97663c7f77c-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:19.707782 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:19.707788    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e1c45a3e-cc55-4400-99ea-f97663c7f77c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:20.331773 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.331739    2581 generic.go:356] "Generic (PLEG): container finished" podID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerID="441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a" exitCode=0
Jul 28 01:48:20.332230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.331792    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"
Jul 28 01:48:20.332230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.331824    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerDied","Data":"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"}
Jul 28 01:48:20.332230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.331884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz" event={"ID":"e1c45a3e-cc55-4400-99ea-f97663c7f77c","Type":"ContainerDied","Data":"f5e67033a5457ca0694427fad4d5048ab9a12b9c545c17a2b60f573dd9b2cef7"}
Jul 28 01:48:20.332230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.331913    2581 scope.go:117] "RemoveContainer" containerID="a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"
Jul 28 01:48:20.340495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.340477    2581 scope.go:117] "RemoveContainer" containerID="441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"
Jul 28 01:48:20.348285 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.348251    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:48:20.349624 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.348907    2581 scope.go:117] "RemoveContainer" containerID="a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"
Jul 28 01:48:20.349624 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:20.349337    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269\": container with ID starting with a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269 not found: ID does not exist" containerID="a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"
Jul 28 01:48:20.349624 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.349375    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269"} err="failed to get container status \"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269\": rpc error: code = NotFound desc = could not find container \"a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269\": container with ID starting with a4d70fbe22445ac786fdf8c7ad220be6e0d9ab751f24a3c0fcdc48f7910b2269 not found: ID does not exist"
Jul 28 01:48:20.349624 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.349399    2581 scope.go:117] "RemoveContainer" containerID="441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"
Jul 28 01:48:20.349829 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:20.349735    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a\": container with ID starting with 441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a not found: ID does not exist" containerID="441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"
Jul 28 01:48:20.349829 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.349767    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a"} err="failed to get container status \"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a\": rpc error: code = NotFound desc = could not find container \"441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a\": container with ID starting with 441490a425267fd785f3b76ffbe09e99c22982cf18d4b33d4178334ce71a7a2a not found: ID does not exist"
Jul 28 01:48:20.350876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:20.350856    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-02c81-predictor-5b98d76d87-zqtqz"]
Jul 28 01:48:21.649502 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:21.649467    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" path="/var/lib/kubelet/pods/e1c45a3e-cc55-4400-99ea-f97663c7f77c/volumes"
Jul 28 01:48:22.103995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:22.103960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:48:22.273881 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:22.273841    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:24.330746 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:24.330717    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:48:24.331247 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:24.331214    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:27.274239 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:27.274201    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:27.274776 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:27.274320    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:32.273915 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:32.273874    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:34.331398 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:34.331348    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:35.353503 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353461    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353815    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container"
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353830    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container"
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353857    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kube-rbac-proxy"
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353865    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kube-rbac-proxy"
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kserve-container"
Jul 28 01:48:35.353937 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.353935    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1c45a3e-cc55-4400-99ea-f97663c7f77c" containerName="kube-rbac-proxy"
Jul 28 01:48:35.358575 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.358551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.360760 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.360733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2a946-serving-cert\""
Jul 28 01:48:35.360900 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.360733    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-2a946-kube-rbac-proxy-sar-config\""
Jul 28 01:48:35.366849 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.366820    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:48:35.371447 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.371416    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:35.430269 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.430225    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.430466 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.430277    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.531179 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.531140    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.531371 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.531191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.531371 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:35.531285    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/sequence-graph-2a946-serving-cert: secret "sequence-graph-2a946-serving-cert" not found
Jul 28 01:48:35.531371 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:35.531351    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls podName:24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9 nodeName:}" failed. No retries permitted until 2026-07-28 01:48:36.031333473 +0000 UTC m=+1364.936902849 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls") pod "sequence-graph-2a946-76d676585d-c76pf" (UID: "24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9") : secret "sequence-graph-2a946-serving-cert" not found
Jul 28 01:48:35.531832 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.531793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:35.627016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:35.626913    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:36.035428 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:36.035375    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:36.037878 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:36.037842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") pod \"sequence-graph-2a946-76d676585d-c76pf\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") " pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:36.269102 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:36.269055    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:36.406702 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:36.406459    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:48:36.409238 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:48:36.409208    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24a3f6c4_a95d_4396_b0d2_7e9a93dd37c9.slice/crio-9e5c69cf97e9ad1583e9153306b7cece225adad929f6b5d0a38dc85084161a5b WatchSource:0}: Error finding container 9e5c69cf97e9ad1583e9153306b7cece225adad929f6b5d0a38dc85084161a5b: Status 404 returned error can't find the container with id 9e5c69cf97e9ad1583e9153306b7cece225adad929f6b5d0a38dc85084161a5b
Jul 28 01:48:37.274612 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:37.274569    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:37.406600 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:37.406560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" event={"ID":"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9","Type":"ContainerStarted","Data":"397ac10c86ae6c9c5978ef3b28a1f4b134da9ad1f8731b88dd4ee76271157eb6"}
Jul 28 01:48:37.406786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:37.406618    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" event={"ID":"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9","Type":"ContainerStarted","Data":"9e5c69cf97e9ad1583e9153306b7cece225adad929f6b5d0a38dc85084161a5b"}
Jul 28 01:48:37.406786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:37.406643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:37.424930 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:37.424870    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podStartSLOduration=2.42485113 podStartE2EDuration="2.42485113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:37.422303238 +0000 UTC m=+1366.327872635" watchObservedRunningTime="2026-07-28 01:48:37.42485113 +0000 UTC m=+1366.330420528"
Jul 28 01:48:42.274498 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:42.274435    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:43.416619 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:43.416587    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:48:44.331460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:44.331416    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:45.455429 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.455386    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:48:45.455938 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.455612    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" containerID="cri-o://397ac10c86ae6c9c5978ef3b28a1f4b134da9ad1f8731b88dd4ee76271157eb6" gracePeriod=30
Jul 28 01:48:45.467191 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.467159    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.571370 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.571337    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:48:45.571644 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.571614    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container" containerID="cri-o://0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb" gracePeriod=30
Jul 28 01:48:45.571756 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.571654    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kube-rbac-proxy" containerID="cri-o://3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35" gracePeriod=30
Jul 28 01:48:45.615629 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.615552    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.641858 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.641821    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:48:45.645664 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.645633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.648317 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.648287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-920f8-predictor-serving-cert\""
Jul 28 01:48:45.648469 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.648294    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-920f8-kube-rbac-proxy-sar-config\""
Jul 28 01:48:45.656889 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.656849    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:48:45.714250 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.714215    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.714421 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.714267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.714421 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.714352    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dlxh\" (UniqueName: \"kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.714621 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.714590    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.810106 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.810070    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.815272 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.815233    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.815439 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.815281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.815439 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.815344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dlxh\" (UniqueName: \"kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.815439 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:45.815394    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-920f8-predictor-serving-cert: secret "success-200-isvc-920f8-predictor-serving-cert" not found
Jul 28 01:48:45.815610 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:45.815461    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls podName:1b0ce3af-9c29-4097-a772-2f9482b07115 nodeName:}" failed. No retries permitted until 2026-07-28 01:48:46.315437764 +0000 UTC m=+1375.221007159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls") pod "success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" (UID: "1b0ce3af-9c29-4097-a772-2f9482b07115") : secret "success-200-isvc-920f8-predictor-serving-cert" not found
Jul 28 01:48:45.816068 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.816045    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.827291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.827264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dlxh\" (UniqueName: \"kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:45.865578 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.865555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:45.916176 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.916139    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls\") pod \"d0503ca4-cb66-4d74-81a9-77e30998df69\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") "
Jul 28 01:48:45.916377 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.916199    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle\") pod \"d0503ca4-cb66-4d74-81a9-77e30998df69\" (UID: \"d0503ca4-cb66-4d74-81a9-77e30998df69\") "
Jul 28 01:48:45.916614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.916586    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d0503ca4-cb66-4d74-81a9-77e30998df69" (UID: "d0503ca4-cb66-4d74-81a9-77e30998df69"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:45.917540 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.917517    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:45.918395 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:45.918373    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0503ca4-cb66-4d74-81a9-77e30998df69" (UID: "d0503ca4-cb66-4d74-81a9-77e30998df69"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:46.017282 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.017246    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0503ca4-cb66-4d74-81a9-77e30998df69-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:46.017282 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.017278    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d0503ca4-cb66-4d74-81a9-77e30998df69-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:46.046521 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.046492    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:46.320115 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.320020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:46.322501 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.322470    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") pod \"success-200-isvc-920f8-predictor-585f7cfd77-h5mg6\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") " pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:46.448109 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.448066    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerID="3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35" exitCode=2
Jul 28 01:48:46.448287 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.448138    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerDied","Data":"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"}
Jul 28 01:48:46.449350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.449330    2581 generic.go:356] "Generic (PLEG): container finished" podID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerID="4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc" exitCode=0
Jul 28 01:48:46.449453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.449364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" event={"ID":"d0503ca4-cb66-4d74-81a9-77e30998df69","Type":"ContainerDied","Data":"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"}
Jul 28 01:48:46.449453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.449378    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"
Jul 28 01:48:46.449453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.449397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9" event={"ID":"d0503ca4-cb66-4d74-81a9-77e30998df69","Type":"ContainerDied","Data":"54c62f0701af119f701473d132d77b24316111b0fae96ca410035eb004cb3e80"}
Jul 28 01:48:46.449453 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.449419    2581 scope.go:117] "RemoveContainer" containerID="4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"
Jul 28 01:48:46.458360 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.458223    2581 scope.go:117] "RemoveContainer" containerID="4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"
Jul 28 01:48:46.458607 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:46.458521    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc\": container with ID starting with 4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc not found: ID does not exist" containerID="4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"
Jul 28 01:48:46.458607 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.458548    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc"} err="failed to get container status \"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc\": rpc error: code = NotFound desc = could not find container \"4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc\": container with ID starting with 4719fd19ebccf486f1bd269939788c38babd9a528f2d3176bc0545e5e777c9dc not found: ID does not exist"
Jul 28 01:48:46.471018 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.470984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:46.477419 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.477388    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-02c81-7c5f497f56-bxss9"]
Jul 28 01:48:46.557194 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.557152    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:46.705167 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:46.705130    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:48:46.708313 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:48:46.708286    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b0ce3af_9c29_4097_a772_2f9482b07115.slice/crio-21700e1a307c3a13de56f0330996411ed4585219c5a94c8d4b9e5656b012831d WatchSource:0}: Error finding container 21700e1a307c3a13de56f0330996411ed4585219c5a94c8d4b9e5656b012831d: Status 404 returned error can't find the container with id 21700e1a307c3a13de56f0330996411ed4585219c5a94c8d4b9e5656b012831d
Jul 28 01:48:47.098119 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.098070    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 01:48:47.464577 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.464475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerStarted","Data":"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"}
Jul 28 01:48:47.465044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.464731    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:47.465044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.464781    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:47.465044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.464827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerStarted","Data":"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"}
Jul 28 01:48:47.465044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.464851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerStarted","Data":"21700e1a307c3a13de56f0330996411ed4585219c5a94c8d4b9e5656b012831d"}
Jul 28 01:48:47.466385 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.466354    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:48:47.484485 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.484426    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podStartSLOduration=2.484410163 podStartE2EDuration="2.484410163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:48:47.483161505 +0000 UTC m=+1376.388730911" watchObservedRunningTime="2026-07-28 01:48:47.484410163 +0000 UTC m=+1376.389979594"
Jul 28 01:48:47.647268 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:47.647232    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" path="/var/lib/kubelet/pods/d0503ca4-cb66-4d74-81a9-77e30998df69/volumes"
Jul 28 01:48:48.414660 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:48.414620    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:48.469047 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:48.469009    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:48:49.223366 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.223333    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:48:49.349093 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.349000    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config\") pod \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") "
Jul 28 01:48:49.349093 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.349040    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vx54g\" (UniqueName: \"kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g\") pod \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") "
Jul 28 01:48:49.349093 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.349091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls\") pod \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\" (UID: \"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7\") "
Jul 28 01:48:49.349428 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.349407    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" (UID: "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:48:49.351264 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.351229    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" (UID: "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:48:49.351340 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.351244    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g" (OuterVolumeSpecName: "kube-api-access-vx54g") pod "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" (UID: "5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7"). InnerVolumeSpecName "kube-api-access-vx54g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:48:49.450153 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.450112    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.450153 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.450144    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vx54g\" (UniqueName: \"kubernetes.io/projected/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-kube-api-access-vx54g\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.450153 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.450155    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:48:49.474746 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.474717    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerID="0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb" exitCode=0
Jul 28 01:48:49.474746 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.474753    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerDied","Data":"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"}
Jul 28 01:48:49.475283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.474776    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"
Jul 28 01:48:49.475283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.474817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l" event={"ID":"5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7","Type":"ContainerDied","Data":"21437a35e190c8e17d4c49239029de92fd01c3e0e526db2d440d1253c34cf128"}
Jul 28 01:48:49.475283 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.474848    2581 scope.go:117] "RemoveContainer" containerID="3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"
Jul 28 01:48:49.483978 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.483959    2581 scope.go:117] "RemoveContainer" containerID="0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"
Jul 28 01:48:49.491908 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.491890    2581 scope.go:117] "RemoveContainer" containerID="3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"
Jul 28 01:48:49.492238 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:49.492219    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35\": container with ID starting with 3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35 not found: ID does not exist" containerID="3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"
Jul 28 01:48:49.492299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.492248    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35"} err="failed to get container status \"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35\": rpc error: code = NotFound desc = could not find container \"3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35\": container with ID starting with 3b760e9b1266fa95150fa7936e68f2508fe70352365327ae01125024b8abfd35 not found: ID does not exist"
Jul 28 01:48:49.492299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.492267    2581 scope.go:117] "RemoveContainer" containerID="0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"
Jul 28 01:48:49.492533 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:48:49.492513    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb\": container with ID starting with 0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb not found: ID does not exist" containerID="0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"
Jul 28 01:48:49.492569 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.492539    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb"} err="failed to get container status \"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb\": rpc error: code = NotFound desc = could not find container \"0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb\": container with ID starting with 0d6b347d550715a5370ecfbfad46ca4d153c95e7c753f8ba0533f411c500a3eb not found: ID does not exist"
Jul 28 01:48:49.497468 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.497440    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:48:49.503581 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.503556    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-2a946-predictor-86c64754c9-psq5l"]
Jul 28 01:48:49.648459 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:49.648380    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" path="/var/lib/kubelet/pods/5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7/volumes"
Jul 28 01:48:53.414750 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:53.414705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:53.473964 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:53.473939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:48:53.474342 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:53.474315    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:48:54.331853 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:54.331788    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:48:57.262852 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:57.262816    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:48:58.415854 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:58.415787    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:48:58.416342 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:48:58.415925    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:49:03.415465 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:03.415422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:49:03.475062 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:03.475023    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:49:04.331931 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:04.331889    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 01:49:08.415177 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:08.415133    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:49:13.414878 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:13.414836    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:49:13.475024 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:13.474979    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:49:14.331815 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:14.331766    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:49:15.589707 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.589666    2581 generic.go:356] "Generic (PLEG): container finished" podID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerID="397ac10c86ae6c9c5978ef3b28a1f4b134da9ad1f8731b88dd4ee76271157eb6" exitCode=0
Jul 28 01:49:15.590189 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.589731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" event={"ID":"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9","Type":"ContainerDied","Data":"397ac10c86ae6c9c5978ef3b28a1f4b134da9ad1f8731b88dd4ee76271157eb6"}
Jul 28 01:49:15.646678 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.646645    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:49:15.780627 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.780595    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") pod \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") "
Jul 28 01:49:15.780834 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.780641    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle\") pod \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\" (UID: \"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9\") "
Jul 28 01:49:15.781057 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.781022    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" (UID: "24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:49:15.782679 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.782659    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" (UID: "24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:49:15.882062 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.882022    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:49:15.882062 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:15.882057    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:49:16.595460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:16.595430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"
Jul 28 01:49:16.595460 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:16.595443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf" event={"ID":"24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9","Type":"ContainerDied","Data":"9e5c69cf97e9ad1583e9153306b7cece225adad929f6b5d0a38dc85084161a5b"}
Jul 28 01:49:16.595972 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:16.595530    2581 scope.go:117] "RemoveContainer" containerID="397ac10c86ae6c9c5978ef3b28a1f4b134da9ad1f8731b88dd4ee76271157eb6"
Jul 28 01:49:16.616920 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:16.616887    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:49:16.621144 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:16.621117    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-2a946-76d676585d-c76pf"]
Jul 28 01:49:17.647013 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:17.646978    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" path="/var/lib/kubelet/pods/24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9/volumes"
Jul 28 01:49:23.474418 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:23.474375    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:49:25.957379 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957340    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957670    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957680    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957694    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kube-rbac-proxy"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957700    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kube-rbac-proxy"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957711    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957717    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957732    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957737    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81"
Jul 28 01:49:25.957778 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kserve-container"
Jul 28 01:49:25.958088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957790    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b8eaec4-835a-43cf-b192-aa4bbe5ef0e7" containerName="kube-rbac-proxy"
Jul 28 01:49:25.958088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957812    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0503ca4-cb66-4d74-81a9-77e30998df69" containerName="ensemble-graph-02c81"
Jul 28 01:49:25.958088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.957820    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24a3f6c4-a95d-4396-b0d2-7e9a93dd37c9" containerName="sequence-graph-2a946"
Jul 28 01:49:25.960869 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.960844    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:25.963357 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.963332    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5a4a0-serving-cert\""
Jul 28 01:49:25.963568 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.963359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"ensemble-graph-5a4a0-kube-rbac-proxy-sar-config\""
Jul 28 01:49:25.963739 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.963720    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:25.969781 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:25.969757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:49:26.074016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.073978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.074233 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.074104    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.175374 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.175327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.175595 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.175392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.175595 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:49:26.175496    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/ensemble-graph-5a4a0-serving-cert: secret "ensemble-graph-5a4a0-serving-cert" not found
Jul 28 01:49:26.175595 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:49:26.175574    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls podName:d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece nodeName:}" failed. No retries permitted until 2026-07-28 01:49:26.67555545 +0000 UTC m=+1415.581124841 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls") pod "ensemble-graph-5a4a0-995768857-2rw89" (UID: "d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece") : secret "ensemble-graph-5a4a0-serving-cert" not found
Jul 28 01:49:26.176136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.176112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.349754 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.349724    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:26.678951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.678850    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.681423 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.681398    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") pod \"ensemble-graph-5a4a0-995768857-2rw89\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") " pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:26.872943 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:26.872899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:27.012204 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:27.012176    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:49:27.014512 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:49:27.014484    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12f1df4_84a9_4ad8_a8c2_c0fbbaca4ece.slice/crio-e66cc5565567f7fdf03d20c530a62962b93c2624dfa27e92be4f159764322c1b WatchSource:0}: Error finding container e66cc5565567f7fdf03d20c530a62962b93c2624dfa27e92be4f159764322c1b: Status 404 returned error can't find the container with id e66cc5565567f7fdf03d20c530a62962b93c2624dfa27e92be4f159764322c1b
Jul 28 01:49:27.646342 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:27.646302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" event={"ID":"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece","Type":"ContainerStarted","Data":"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"}
Jul 28 01:49:27.646552 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:27.646361    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:27.646552 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:27.646382    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" event={"ID":"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece","Type":"ContainerStarted","Data":"e66cc5565567f7fdf03d20c530a62962b93c2624dfa27e92be4f159764322c1b"}
Jul 28 01:49:27.658962 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:27.658903    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podStartSLOduration=2.658885327 podStartE2EDuration="2.658885327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:49:27.65774154 +0000 UTC m=+1416.563310938" watchObservedRunningTime="2026-07-28 01:49:27.658885327 +0000 UTC m=+1416.564454725"
Jul 28 01:49:33.475293 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:33.475253    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:49:33.652376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:33.652350    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:49:43.475021 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:43.474940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:49:55.687954 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.687918    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:49:55.692307 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.692285    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.694554 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.694536    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-920f8-kube-rbac-proxy-sar-config\""
Jul 28 01:49:55.694666 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.694537    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"sequence-graph-920f8-serving-cert\""
Jul 28 01:49:55.700786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.700757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:49:55.819378 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.819340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.819572 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.819455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.920205 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.920159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.920399 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.920218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.920909 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.920885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:55.922574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:55.922556    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls\") pod \"sequence-graph-920f8-67d69d5c5b-bmzql\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") " pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:56.003512 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.003486    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:56.134732 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.134702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:49:56.137253 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:49:56.137215    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1f082b_5467_4f38_bf69_ebcce81b199a.slice/crio-eec0bd7d55631517c40f1ec48e3a54bd3d30994c3e6574fde9249b404be9a65c WatchSource:0}: Error finding container eec0bd7d55631517c40f1ec48e3a54bd3d30994c3e6574fde9249b404be9a65c: Status 404 returned error can't find the container with id eec0bd7d55631517c40f1ec48e3a54bd3d30994c3e6574fde9249b404be9a65c
Jul 28 01:49:56.346775 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.346686    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:56.404711 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.404680    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:49:56.771591 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.771557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" event={"ID":"5b1f082b-5467-4f38-bf69-ebcce81b199a","Type":"ContainerStarted","Data":"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"}
Jul 28 01:49:56.772015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.771610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" event={"ID":"5b1f082b-5467-4f38-bf69-ebcce81b199a","Type":"ContainerStarted","Data":"eec0bd7d55631517c40f1ec48e3a54bd3d30994c3e6574fde9249b404be9a65c"}
Jul 28 01:49:56.772015 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.771632    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:49:56.795218 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:49:56.795168    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podStartSLOduration=1.795151552 podStartE2EDuration="1.795151552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:49:56.793264663 +0000 UTC m=+1445.698834073" watchObservedRunningTime="2026-07-28 01:49:56.795151552 +0000 UTC m=+1445.700720950"
Jul 28 01:50:02.781144 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:50:02.781118    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:50:45.693017 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:50:45.692977    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:50:51.656555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:50:51.656524    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:50:51.659291 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:50:51.659266    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:55:51.681549 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:55:51.681517    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:55:51.685702 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:55:51.685674    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 01:56:21.150825 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:56:21.150760    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:37.617436 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:37.617336    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:40.507194 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.507161    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:57:40.507596 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.507362    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" containerID="cri-o://936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93" gracePeriod=30
Jul 28 01:57:40.556527 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.556488    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:40.738620 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.738583    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:57:40.738901 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.738873    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container" containerID="cri-o://9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214" gracePeriod=30
Jul 28 01:57:40.739008 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.738923    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kube-rbac-proxy" containerID="cri-o://8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e" gracePeriod=30
Jul 28 01:57:40.745122 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.745094    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:40.784417 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.784335    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:57:40.787912 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.787889    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.790208 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.790184    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bc47b-predictor-serving-cert\""
Jul 28 01:57:40.790321 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.790187    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-bc47b-kube-rbac-proxy-sar-config\""
Jul 28 01:57:40.799199 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.799171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:57:40.827954 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.827917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.828162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.827976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.828162 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.828037    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stxl\" (UniqueName: \"kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.841084 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.841042    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:40.924614 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.924577    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:40.929564 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.929530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.929747 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.929606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.929747 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.929656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stxl\" (UniqueName: \"kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.930356 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.930333    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.932286 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.932258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:40.938792 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:40.938763    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stxl\" (UniqueName: \"kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl\") pod \"success-200-isvc-bc47b-predictor-766766c867-7h5fm\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") " pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:41.027487 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.027452    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:41.098652 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.098558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:41.256951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.256917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:57:41.260493 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:57:41.260466    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef7ac4cc_6535_46cc_b04a_7b376d52ef01.slice/crio-b924f3ad20658ed9ca03cf2da772b26c456ded4dd344e814938a8c920f14c690 WatchSource:0}: Error finding container b924f3ad20658ed9ca03cf2da772b26c456ded4dd344e814938a8c920f14c690: Status 404 returned error can't find the container with id b924f3ad20658ed9ca03cf2da772b26c456ded4dd344e814938a8c920f14c690
Jul 28 01:57:41.277080 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.277050    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:57:41.635574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.635475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerStarted","Data":"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"}
Jul 28 01:57:41.635574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.635557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerStarted","Data":"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"}
Jul 28 01:57:41.636090 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.635590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerStarted","Data":"b924f3ad20658ed9ca03cf2da772b26c456ded4dd344e814938a8c920f14c690"}
Jul 28 01:57:41.636158 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.636113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:41.636336 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.636317    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:41.638260 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.638138    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:57:41.638260 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.638231    2581 generic.go:356] "Generic (PLEG): container finished" podID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerID="8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e" exitCode=2
Jul 28 01:57:41.638420 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.638291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerDied","Data":"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"}
Jul 28 01:57:41.656640 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:41.656578    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podStartSLOduration=1.656559544 podStartE2EDuration="1.656559544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:57:41.656456967 +0000 UTC m=+1910.562026369" watchObservedRunningTime="2026-07-28 01:57:41.656559544 +0000 UTC m=+1910.562128942"
Jul 28 01:57:42.643932 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:42.643894    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:57:43.647865 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:43.647823    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:57:43.650023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:43.649998    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:57:44.190379 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.190354    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:57:44.258742 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.258710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntg6k\" (UniqueName: \"kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k\") pod \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") "
Jul 28 01:57:44.258742 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.258749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config\") pod \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") "
Jul 28 01:57:44.259000 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.258808    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") pod \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\" (UID: \"a1cacbd6-d19e-4f79-99af-ea1ce733efc7\") "
Jul 28 01:57:44.259110 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.259086    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1cacbd6-d19e-4f79-99af-ea1ce733efc7" (UID: "a1cacbd6-d19e-4f79-99af-ea1ce733efc7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:57:44.260989 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.260956    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1cacbd6-d19e-4f79-99af-ea1ce733efc7" (UID: "a1cacbd6-d19e-4f79-99af-ea1ce733efc7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:57:44.261103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.260964    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k" (OuterVolumeSpecName: "kube-api-access-ntg6k") pod "a1cacbd6-d19e-4f79-99af-ea1ce733efc7" (UID: "a1cacbd6-d19e-4f79-99af-ea1ce733efc7"). InnerVolumeSpecName "kube-api-access-ntg6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:57:44.359639 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.359540    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntg6k\" (UniqueName: \"kubernetes.io/projected/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-api-access-ntg6k\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.359639 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.359575    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.359639 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.359589    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1cacbd6-d19e-4f79-99af-ea1ce733efc7-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:57:44.653376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.653279    2581 generic.go:356] "Generic (PLEG): container finished" podID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerID="9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214" exitCode=0
Jul 28 01:57:44.653376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.653336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"
Jul 28 01:57:44.653376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.653351    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerDied","Data":"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"}
Jul 28 01:57:44.654023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.653410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh" event={"ID":"a1cacbd6-d19e-4f79-99af-ea1ce733efc7","Type":"ContainerDied","Data":"0bba37df57e18a456bd72294cd882153781b6fb05b39ced1892dbb74e02d3844"}
Jul 28 01:57:44.654023 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.653438    2581 scope.go:117] "RemoveContainer" containerID="8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"
Jul 28 01:57:44.661833 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.661813    2581 scope.go:117] "RemoveContainer" containerID="9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"
Jul 28 01:57:44.670734 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.670710    2581 scope.go:117] "RemoveContainer" containerID="8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"
Jul 28 01:57:44.671081 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:57:44.671061    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e\": container with ID starting with 8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e not found: ID does not exist" containerID="8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"
Jul 28 01:57:44.671163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.671092    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e"} err="failed to get container status \"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e\": rpc error: code = NotFound desc = could not find container \"8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e\": container with ID starting with 8cb9c3e54d60055c53d9fe0ef1bf7df052ad96fca84657c7c8d1fb31a36d134e not found: ID does not exist"
Jul 28 01:57:44.671163 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.671111    2581 scope.go:117] "RemoveContainer" containerID="9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"
Jul 28 01:57:44.671425 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:57:44.671406    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214\": container with ID starting with 9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214 not found: ID does not exist" containerID="9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"
Jul 28 01:57:44.671483 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.671432    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214"} err="failed to get container status \"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214\": rpc error: code = NotFound desc = could not find container \"9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214\": container with ID starting with 9b3a13b015dc66c3547496923e99cebe853e400b844de89de2836a4b5c94e214 not found: ID does not exist"
Jul 28 01:57:44.675080 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.675054    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:57:44.679531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:44.679509    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-5a4a0-predictor-777c9d8757-wdkqh"]
Jul 28 01:57:45.648148 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:45.648117    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" path="/var/lib/kubelet/pods/a1cacbd6-d19e-4f79-99af-ea1ce733efc7/volumes"
Jul 28 01:57:48.650993 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:48.650950    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:57:48.652004 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:48.651985    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:57:48.652444 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:48.652422    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:57:53.651100 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:53.651058    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:57:53.651574 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:53.651195    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:57:58.651819 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:58.651753    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:57:58.654306 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:57:58.652413    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:58:03.650130 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:03.650091    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:08.651118 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:08.651073    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:08.652443 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:08.652419    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:58:10.169664 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.169628    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:58:10.170091 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.169849    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" containerID="cri-o://27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9" gracePeriod=30
Jul 28 01:58:10.188810 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.188773    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.278866 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.278825    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:58:10.279126 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.279067    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" containerID="cri-o://5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0" gracePeriod=30
Jul 28 01:58:10.279193 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.279130    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kube-rbac-proxy" containerID="cri-o://906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712" gracePeriod=30
Jul 28 01:58:10.310379 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.310342    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.351694 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.351653    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 01:58:10.352048 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352032    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kube-rbac-proxy"
Jul 28 01:58:10.352103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kube-rbac-proxy"
Jul 28 01:58:10.352103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352068    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container"
Jul 28 01:58:10.352103 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352074    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container"
Jul 28 01:58:10.352202 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352123    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kube-rbac-proxy"
Jul 28 01:58:10.352202 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.352133    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1cacbd6-d19e-4f79-99af-ea1ce733efc7" containerName="kserve-container"
Jul 28 01:58:10.355353 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.355333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.357623 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.357593    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-704b9-predictor-serving-cert\""
Jul 28 01:58:10.357755 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.357602    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-704b9-kube-rbac-proxy-sar-config\""
Jul 28 01:58:10.366197 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.366171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 01:58:10.404468 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.404439    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.484686 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.484594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.484686 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.484661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxsfv\" (UniqueName: \"kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.484892 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.484706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.512116 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.512073    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.585315 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.585284    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.585495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.585338    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxsfv\" (UniqueName: \"kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.585495 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.585382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.585495 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:10.585460    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-704b9-predictor-serving-cert: secret "success-200-isvc-704b9-predictor-serving-cert" not found
Jul 28 01:58:10.585664 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:10.585551    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls podName:8b088f90-c19d-4e15-acf4-210df4ecf4b4 nodeName:}" failed. No retries permitted until 2026-07-28 01:58:11.085527811 +0000 UTC m=+1939.991097186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls") pod "success-200-isvc-704b9-predictor-5774cf759-k29gm" (UID: "8b088f90-c19d-4e15-acf4-210df4ecf4b4") : secret "success-200-isvc-704b9-predictor-serving-cert" not found
Jul 28 01:58:10.586185 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.586168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.596327 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.596296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxsfv\" (UniqueName: \"kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:10.658207 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.658179    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:58:10.676130 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.676096    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.764541 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.764507    2581 generic.go:356] "Generic (PLEG): container finished" podID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerID="936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93" exitCode=0
Jul 28 01:58:10.764734 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.764551    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"
Jul 28 01:58:10.764734 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.764597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" event={"ID":"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece","Type":"ContainerDied","Data":"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"}
Jul 28 01:58:10.764734 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.764648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89" event={"ID":"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece","Type":"ContainerDied","Data":"e66cc5565567f7fdf03d20c530a62962b93c2624dfa27e92be4f159764322c1b"}
Jul 28 01:58:10.764734 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.764674    2581 scope.go:117] "RemoveContainer" containerID="936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"
Jul 28 01:58:10.766470 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.766450    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerID="906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712" exitCode=2
Jul 28 01:58:10.766566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.766481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerDied","Data":"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"}
Jul 28 01:58:10.772828 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.772807    2581 scope.go:117] "RemoveContainer" containerID="936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"
Jul 28 01:58:10.773094 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:10.773076    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93\": container with ID starting with 936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93 not found: ID does not exist" containerID="936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"
Jul 28 01:58:10.773156 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.773102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93"} err="failed to get container status \"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93\": rpc error: code = NotFound desc = could not find container \"936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93\": container with ID starting with 936f21ba98c282d9d35d2bcb38f5d1903be16626bcc31eff4f88a6b9766a6a93 not found: ID does not exist"
Jul 28 01:58:10.773935 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.773915    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:10.787065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.787042    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle\") pod \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") "
Jul 28 01:58:10.787174 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.787082    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") pod \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\" (UID: \"d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece\") "
Jul 28 01:58:10.787380 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.787361    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" (UID: "d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:10.789161 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.789137    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" (UID: "d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:10.887668 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.887631    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:10.887668 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:10.887658    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:11.086088 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.086056    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:58:11.089787 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.089761    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:11.090032 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.090011    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/ensemble-graph-5a4a0-995768857-2rw89"]
Jul 28 01:58:11.092212 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.092191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") pod \"success-200-isvc-704b9-predictor-5774cf759-k29gm\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") " pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:11.265894 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.265860    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:11.404054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.404029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 01:58:11.406110 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:58:11.406081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b088f90_c19d_4e15_acf4_210df4ecf4b4.slice/crio-54e31fc2e298ce60e92f983b35ebd840ca7203410bfc110485625b5412e68391 WatchSource:0}: Error finding container 54e31fc2e298ce60e92f983b35ebd840ca7203410bfc110485625b5412e68391: Status 404 returned error can't find the container with id 54e31fc2e298ce60e92f983b35ebd840ca7203410bfc110485625b5412e68391
Jul 28 01:58:11.653136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.650091    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" path="/var/lib/kubelet/pods/d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece/volumes"
Jul 28 01:58:11.772646 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.772609    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerStarted","Data":"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"}
Jul 28 01:58:11.772879 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.772677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerStarted","Data":"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"}
Jul 28 01:58:11.772879 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.772831    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:11.773009 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.772988    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:11.773065 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.773029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerStarted","Data":"54e31fc2e298ce60e92f983b35ebd840ca7203410bfc110485625b5412e68391"}
Jul 28 01:58:11.774173 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.774149    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:11.796186 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:11.796134    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podStartSLOduration=1.796116916 podStartE2EDuration="1.796116916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:11.794557574 +0000 UTC m=+1940.700126973" watchObservedRunningTime="2026-07-28 01:58:11.796116916 +0000 UTC m=+1940.701686313"
Jul 28 01:58:12.777660 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:12.777618    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:12.779165 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:12.779136    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:13.470028 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.469989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 01:58:13.475401 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.475370    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 01:58:13.636050 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.636021    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:58:13.711945 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.711856    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config\") pod \"1b0ce3af-9c29-4097-a772-2f9482b07115\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") "
Jul 28 01:58:13.711945 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.711945    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") pod \"1b0ce3af-9c29-4097-a772-2f9482b07115\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") "
Jul 28 01:58:13.712154 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.711980    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dlxh\" (UniqueName: \"kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh\") pod \"1b0ce3af-9c29-4097-a772-2f9482b07115\" (UID: \"1b0ce3af-9c29-4097-a772-2f9482b07115\") "
Jul 28 01:58:13.712308 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.712274    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1b0ce3af-9c29-4097-a772-2f9482b07115" (UID: "1b0ce3af-9c29-4097-a772-2f9482b07115"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:13.714054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.714028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh" (OuterVolumeSpecName: "kube-api-access-5dlxh") pod "1b0ce3af-9c29-4097-a772-2f9482b07115" (UID: "1b0ce3af-9c29-4097-a772-2f9482b07115"). InnerVolumeSpecName "kube-api-access-5dlxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:13.714054 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.714046    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1b0ce3af-9c29-4097-a772-2f9482b07115" (UID: "1b0ce3af-9c29-4097-a772-2f9482b07115"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:13.783246 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.783209    2581 generic.go:356] "Generic (PLEG): container finished" podID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerID="5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0" exitCode=0
Jul 28 01:58:13.783693 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.783265    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"
Jul 28 01:58:13.783693 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.783279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerDied","Data":"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"}
Jul 28 01:58:13.783693 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.783342    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6" event={"ID":"1b0ce3af-9c29-4097-a772-2f9482b07115","Type":"ContainerDied","Data":"21700e1a307c3a13de56f0330996411ed4585219c5a94c8d4b9e5656b012831d"}
Jul 28 01:58:13.783693 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.783379    2581 scope.go:117] "RemoveContainer" containerID="906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"
Jul 28 01:58:13.791995 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.791976    2581 scope.go:117] "RemoveContainer" containerID="5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"
Jul 28 01:58:13.799684 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.799665    2581 scope.go:117] "RemoveContainer" containerID="906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"
Jul 28 01:58:13.799994 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:13.799974    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712\": container with ID starting with 906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712 not found: ID does not exist" containerID="906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"
Jul 28 01:58:13.800076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.800002    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712"} err="failed to get container status \"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712\": rpc error: code = NotFound desc = could not find container \"906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712\": container with ID starting with 906d398cc2481fa535c50e19b9d5ca091baab056b0bee4b2ca695f9837cb8712 not found: ID does not exist"
Jul 28 01:58:13.800076 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.800019    2581 scope.go:117] "RemoveContainer" containerID="5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"
Jul 28 01:58:13.800308 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:13.800294    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0\": container with ID starting with 5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0 not found: ID does not exist" containerID="5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"
Jul 28 01:58:13.800349 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.800315    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0"} err="failed to get container status \"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0\": rpc error: code = NotFound desc = could not find container \"5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0\": container with ID starting with 5552ab248af0952034e0a11337529e6a116e1b58522fd85c811e37ea31c5a1b0 not found: ID does not exist"
Jul 28 01:58:13.806026 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.806001    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:58:13.807590 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.807564    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-920f8-predictor-585f7cfd77-h5mg6"]
Jul 28 01:58:13.813326 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.813302    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1b0ce3af-9c29-4097-a772-2f9482b07115-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:13.813326 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.813322    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dlxh\" (UniqueName: \"kubernetes.io/projected/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-api-access-5dlxh\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:13.813518 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:13.813333    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1b0ce3af-9c29-4097-a772-2f9482b07115-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:15.513153 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:15.513114    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:15.646909 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:15.646874    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" path="/var/lib/kubelet/pods/1b0ce3af-9c29-4097-a772-2f9482b07115/volumes"
Jul 28 01:58:17.779886 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:17.779838    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:17.781664 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:17.781635    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:58:17.782230 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:17.782200    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:18.653027 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:18.652989    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 01:58:22.780025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:22.779983    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:22.780432 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:22.780113    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:58:27.779396 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:27.779351    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:27.782706 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:27.782675    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:28.653548 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:28.653519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:58:32.779370 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:32.779325    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:37.779754 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:37.779705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:37.783043 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:37.783012    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:40.312935 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.312907    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:58:40.322786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.322753    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle\") pod \"5b1f082b-5467-4f38-bf69-ebcce81b199a\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") "
Jul 28 01:58:40.322786 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.322830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls\") pod \"5b1f082b-5467-4f38-bf69-ebcce81b199a\" (UID: \"5b1f082b-5467-4f38-bf69-ebcce81b199a\") "
Jul 28 01:58:40.323157 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.323136    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "5b1f082b-5467-4f38-bf69-ebcce81b199a" (UID: "5b1f082b-5467-4f38-bf69-ebcce81b199a"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:40.324813 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.324780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5b1f082b-5467-4f38-bf69-ebcce81b199a" (UID: "5b1f082b-5467-4f38-bf69-ebcce81b199a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:40.423790 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.423697    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5b1f082b-5467-4f38-bf69-ebcce81b199a-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:40.423790 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.423729    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5b1f082b-5467-4f38-bf69-ebcce81b199a-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:40.785856 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.785820    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:58:40.786198 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786184    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kube-rbac-proxy"
Jul 28 01:58:40.786247 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786200    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kube-rbac-proxy"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786246    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786255    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786268    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786281    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container"
Jul 28 01:58:40.786299 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786286    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container"
Jul 28 01:58:40.786517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786339    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kserve-container"
Jul 28 01:58:40.786517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786349    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b0ce3af-9c29-4097-a772-2f9482b07115" containerName="kube-rbac-proxy"
Jul 28 01:58:40.786517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786355    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerName="sequence-graph-920f8"
Jul 28 01:58:40.786517 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.786361    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12f1df4-84a9-4ad8-a8c2-c0fbbaca4ece" containerName="ensemble-graph-5a4a0"
Jul 28 01:58:40.789404 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.789387    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.791770 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.791728    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-bc47b-serving-cert\""
Jul 28 01:58:40.792024 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.791781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-bc47b-kube-rbac-proxy-sar-config\""
Jul 28 01:58:40.792364 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.792322    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:40.799836 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.799811    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:58:40.827269 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.827232    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.827462 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.827278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.889257 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.889228    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:40.895604 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.895575    2581 generic.go:356] "Generic (PLEG): container finished" podID="5b1f082b-5467-4f38-bf69-ebcce81b199a" containerID="27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9" exitCode=0
Jul 28 01:58:40.895774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.895620    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"
Jul 28 01:58:40.895774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.895661    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" event={"ID":"5b1f082b-5467-4f38-bf69-ebcce81b199a","Type":"ContainerDied","Data":"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"}
Jul 28 01:58:40.895774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.895704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql" event={"ID":"5b1f082b-5467-4f38-bf69-ebcce81b199a","Type":"ContainerDied","Data":"eec0bd7d55631517c40f1ec48e3a54bd3d30994c3e6574fde9249b404be9a65c"}
Jul 28 01:58:40.895774 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.895731    2581 scope.go:117] "RemoveContainer" containerID="27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"
Jul 28 01:58:40.903868 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.903848    2581 scope.go:117] "RemoveContainer" containerID="27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"
Jul 28 01:58:40.904148 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:40.904128    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9\": container with ID starting with 27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9 not found: ID does not exist" containerID="27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"
Jul 28 01:58:40.904219 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.904157    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9"} err="failed to get container status \"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9\": rpc error: code = NotFound desc = could not find container \"27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9\": container with ID starting with 27486cc2f8a6a14c2fc92409e134290cb8848917db813b31b51d4af2bafb30d9 not found: ID does not exist"
Jul 28 01:58:40.917240 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.917212    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:58:40.921815 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.921779    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/sequence-graph-920f8-67d69d5c5b-bmzql"]
Jul 28 01:58:40.928295 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.928267    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.928419 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.928305    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.928895 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.928872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:40.930609 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:40.930588    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls\") pod \"splitter-graph-bc47b-5b5f84d749-r4bth\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") " pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:41.100992 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.100901    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:41.235397 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.235361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:58:41.239105 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:58:41.239076    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e0c0a6_630f_474c_8d27_bca84a9d28f7.slice/crio-2c3ce08aa35a52f875776f9400c8bfc1c3a3b5b14981d6bd0c591e4b6ad7ec76 WatchSource:0}: Error finding container 2c3ce08aa35a52f875776f9400c8bfc1c3a3b5b14981d6bd0c591e4b6ad7ec76: Status 404 returned error can't find the container with id 2c3ce08aa35a52f875776f9400c8bfc1c3a3b5b14981d6bd0c591e4b6ad7ec76
Jul 28 01:58:41.647122 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.647088    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b1f082b-5467-4f38-bf69-ebcce81b199a" path="/var/lib/kubelet/pods/5b1f082b-5467-4f38-bf69-ebcce81b199a/volumes"
Jul 28 01:58:41.901273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.901169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" event={"ID":"77e0c0a6-630f-474c-8d27-bca84a9d28f7","Type":"ContainerStarted","Data":"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"}
Jul 28 01:58:41.901273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.901223    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:41.901273 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.901239    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" event={"ID":"77e0c0a6-630f-474c-8d27-bca84a9d28f7","Type":"ContainerStarted","Data":"2c3ce08aa35a52f875776f9400c8bfc1c3a3b5b14981d6bd0c591e4b6ad7ec76"}
Jul 28 01:58:41.920254 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:41.920205    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podStartSLOduration=1.9201854040000002 podStartE2EDuration="1.920185404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:41.91973607 +0000 UTC m=+1970.825305468" watchObservedRunningTime="2026-07-28 01:58:41.920185404 +0000 UTC m=+1970.825754803"
Jul 28 01:58:47.782667 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:47.782574    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:47.912665 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:47.912633    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:58:50.851140 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:50.851109    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:58:50.851566 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:50.851309    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" containerID="cri-o://9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948" gracePeriod=30
Jul 28 01:58:50.878402 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:50.878368    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:50.988402 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:50.988366    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:51.015440 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.015407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:58:51.015662 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.015637    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container" containerID="cri-o://ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0" gracePeriod=30
Jul 28 01:58:51.015737 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.015687    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kube-rbac-proxy" containerID="cri-o://e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c" gracePeriod=30
Jul 28 01:58:51.048011 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.047977    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 01:58:51.052771 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.052744    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.055605 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.055572    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7876a-predictor-serving-cert\""
Jul 28 01:58:51.055605 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.055596    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-7876a-kube-rbac-proxy-sar-config\""
Jul 28 01:58:51.061349 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.061326    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 01:58:51.075740 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.075700    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:51.098457 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.098427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.098628 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.098469    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbtjj\" (UniqueName: \"kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.098628 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.098489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.155454 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.155417    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:51.199689 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.199652    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.199886 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.199711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbtjj\" (UniqueName: \"kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.199886 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.199740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.199886 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:51.199832    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-7876a-predictor-serving-cert: secret "success-200-isvc-7876a-predictor-serving-cert" not found
Jul 28 01:58:51.200043 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:51.199915    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls podName:9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad nodeName:}" failed. No retries permitted until 2026-07-28 01:58:51.699897114 +0000 UTC m=+1980.605466490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls") pod "success-200-isvc-7876a-predictor-7b564495c-rrcc7" (UID: "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad") : secret "success-200-isvc-7876a-predictor-serving-cert" not found
Jul 28 01:58:51.200555 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.200534    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.208972 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.208943    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbtjj\" (UniqueName: \"kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.256146 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.256110    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:51.363448 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.363419    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:58:51.703562 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.703520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.706025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.705998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") pod \"success-200-isvc-7876a-predictor-7b564495c-rrcc7\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") " pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:51.950716 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.950668    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerID="e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c" exitCode=2
Jul 28 01:58:51.951136 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.950727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerDied","Data":"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"}
Jul 28 01:58:51.964016 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:51.963928    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:52.103444 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.103408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 01:58:52.106267 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:58:52.106237    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fef3c61_9e7b_4a3c_9c37_f15a06fcf3ad.slice/crio-42e0e51d6f23935002ba1c5043ffd6caf3ee868d15d14fcc02f2cd0800e2959c WatchSource:0}: Error finding container 42e0e51d6f23935002ba1c5043ffd6caf3ee868d15d14fcc02f2cd0800e2959c: Status 404 returned error can't find the container with id 42e0e51d6f23935002ba1c5043ffd6caf3ee868d15d14fcc02f2cd0800e2959c
Jul 28 01:58:52.911024 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.910984    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:52.956139 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.956107    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerStarted","Data":"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"}
Jul 28 01:58:52.956535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.956162    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerStarted","Data":"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"}
Jul 28 01:58:52.956535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.956184    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerStarted","Data":"42e0e51d6f23935002ba1c5043ffd6caf3ee868d15d14fcc02f2cd0800e2959c"}
Jul 28 01:58:52.956535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.956206    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:52.956535 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.956225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:52.957844 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.957816    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:58:52.974917 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:52.974871    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podStartSLOduration=1.974853854 podStartE2EDuration="1.974853854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:58:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:58:52.973389453 +0000 UTC m=+1981.878958904" watchObservedRunningTime="2026-07-28 01:58:52.974853854 +0000 UTC m=+1981.880423317"
Jul 28 01:58:53.648872 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:53.648827    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.36:8643/healthz\": dial tcp 10.134.0.36:8643: connect: connection refused"
Jul 28 01:58:53.960876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:53.960766    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:58:54.464236 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.464209    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:58:54.521735 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.521701    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5stxl\" (UniqueName: \"kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl\") pod \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") "
Jul 28 01:58:54.521927 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.521778    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config\") pod \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") "
Jul 28 01:58:54.521927 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.521832    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls\") pod \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\" (UID: \"ef7ac4cc-6535-46cc-b04a-7b376d52ef01\") "
Jul 28 01:58:54.522164 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.522138    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ef7ac4cc-6535-46cc-b04a-7b376d52ef01" (UID: "ef7ac4cc-6535-46cc-b04a-7b376d52ef01"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:58:54.523872 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.523833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl" (OuterVolumeSpecName: "kube-api-access-5stxl") pod "ef7ac4cc-6535-46cc-b04a-7b376d52ef01" (UID: "ef7ac4cc-6535-46cc-b04a-7b376d52ef01"). InnerVolumeSpecName "kube-api-access-5stxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 01:58:54.523981 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.523930    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ef7ac4cc-6535-46cc-b04a-7b376d52ef01" (UID: "ef7ac4cc-6535-46cc-b04a-7b376d52ef01"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:58:54.622424 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.622384    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5stxl\" (UniqueName: \"kubernetes.io/projected/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-api-access-5stxl\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.622424 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.622416    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.622424 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.622426    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ef7ac4cc-6535-46cc-b04a-7b376d52ef01-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:58:54.965996 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.965909    2581 generic.go:356] "Generic (PLEG): container finished" podID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerID="ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0" exitCode=0
Jul 28 01:58:54.965996 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.965962    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"
Jul 28 01:58:54.966511 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.965999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerDied","Data":"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"}
Jul 28 01:58:54.966511 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.966073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm" event={"ID":"ef7ac4cc-6535-46cc-b04a-7b376d52ef01","Type":"ContainerDied","Data":"b924f3ad20658ed9ca03cf2da772b26c456ded4dd344e814938a8c920f14c690"}
Jul 28 01:58:54.966511 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.966113    2581 scope.go:117] "RemoveContainer" containerID="e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"
Jul 28 01:58:54.974300 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.974283    2581 scope.go:117] "RemoveContainer" containerID="ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"
Jul 28 01:58:54.981641 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.981624    2581 scope.go:117] "RemoveContainer" containerID="e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"
Jul 28 01:58:54.981906 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:54.981887    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c\": container with ID starting with e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c not found: ID does not exist" containerID="e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"
Jul 28 01:58:54.982040 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.981919    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c"} err="failed to get container status \"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c\": rpc error: code = NotFound desc = could not find container \"e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c\": container with ID starting with e309c916e90c0e4cfe84eaab7da2eca4fc980ec637fffe95c382bd84366e4a6c not found: ID does not exist"
Jul 28 01:58:54.982040 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.981942    2581 scope.go:117] "RemoveContainer" containerID="ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"
Jul 28 01:58:54.982181 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:58:54.982165    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0\": container with ID starting with ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0 not found: ID does not exist" containerID="ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"
Jul 28 01:58:54.982228 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.982183    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0"} err="failed to get container status \"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0\": rpc error: code = NotFound desc = could not find container \"ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0\": container with ID starting with ec0339ca745800560add7964361f8d710cf8bdb515dea4b11968df68f17f3fe0 not found: ID does not exist"
Jul 28 01:58:54.986961 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.986938    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:58:54.991962 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:54.991941    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-bc47b-predictor-766766c867-7h5fm"]
Jul 28 01:58:55.647018 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:55.646989    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" path="/var/lib/kubelet/pods/ef7ac4cc-6535-46cc-b04a-7b376d52ef01/volumes"
Jul 28 01:58:57.782440 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:57.782398    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 01:58:57.911116 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:57.911079    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:58:58.965351 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:58.965314    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:58:58.965982 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:58:58.965948    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:59:02.911350 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:02.911306    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:59:02.911741 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:02.911427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:59:07.783491 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:07.783453    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 01:59:07.911523 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:07.911482    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:59:08.966297 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:08.966256    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:59:12.910695 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:12.910655    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:59:17.910998 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:17.910952    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 01:59:18.966682 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:18.966642    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:59:20.437415 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437373    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 01:59:20.437833 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437733    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container"
Jul 28 01:59:20.437833 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437745    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container"
Jul 28 01:59:20.437833 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437763    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kube-rbac-proxy"
Jul 28 01:59:20.437833 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437769    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kube-rbac-proxy"
Jul 28 01:59:20.437985 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437841    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kube-rbac-proxy"
Jul 28 01:59:20.437985 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.437854    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7ac4cc-6535-46cc-b04a-7b376d52ef01" containerName="kserve-container"
Jul 28 01:59:20.442337 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.442318    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.445418 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.445393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-704b9-serving-cert\""
Jul 28 01:59:20.445757 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.445740    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"switch-graph-704b9-kube-rbac-proxy-sar-config\""
Jul 28 01:59:20.451365 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.451333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 01:59:20.457117 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.457088    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:20.541025 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.540985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.541203 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.541106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.641762 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.641723    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.641993 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.641777    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.641993 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:59:20.641974    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/switch-graph-704b9-serving-cert: secret "switch-graph-704b9-serving-cert" not found
Jul 28 01:59:20.642150 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:59:20.642077    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls podName:81f7fa45-630c-432c-bc84-95bdfeabce3b nodeName:}" failed. No retries permitted until 2026-07-28 01:59:21.142054416 +0000 UTC m=+2010.047623794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls") pod "switch-graph-704b9-699d45d87-vr6gx" (UID: "81f7fa45-630c-432c-bc84-95bdfeabce3b") : secret "switch-graph-704b9-serving-cert" not found
Jul 28 01:59:20.642522 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.642500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:20.704184 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.704089    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:20.988971 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:20.988944    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:59:21.045044 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.045005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls\") pod \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") "
Jul 28 01:59:21.045236 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.045052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle\") pod \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\" (UID: \"77e0c0a6-630f-474c-8d27-bca84a9d28f7\") "
Jul 28 01:59:21.045422 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.045387    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "77e0c0a6-630f-474c-8d27-bca84a9d28f7" (UID: "77e0c0a6-630f-474c-8d27-bca84a9d28f7"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 01:59:21.047094 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.047067    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "77e0c0a6-630f-474c-8d27-bca84a9d28f7" (UID: "77e0c0a6-630f-474c-8d27-bca84a9d28f7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 01:59:21.074623 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.074587    2581 generic.go:356] "Generic (PLEG): container finished" podID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerID="9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948" exitCode=0
Jul 28 01:59:21.074817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.074630    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"
Jul 28 01:59:21.074817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.074668    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" event={"ID":"77e0c0a6-630f-474c-8d27-bca84a9d28f7","Type":"ContainerDied","Data":"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"}
Jul 28 01:59:21.074817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.074722    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth" event={"ID":"77e0c0a6-630f-474c-8d27-bca84a9d28f7","Type":"ContainerDied","Data":"2c3ce08aa35a52f875776f9400c8bfc1c3a3b5b14981d6bd0c591e4b6ad7ec76"}
Jul 28 01:59:21.074817 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.074745    2581 scope.go:117] "RemoveContainer" containerID="9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"
Jul 28 01:59:21.083772 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.083755    2581 scope.go:117] "RemoveContainer" containerID="9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"
Jul 28 01:59:21.084093 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:59:21.084076    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948\": container with ID starting with 9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948 not found: ID does not exist" containerID="9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"
Jul 28 01:59:21.084149 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.084102    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948"} err="failed to get container status \"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948\": rpc error: code = NotFound desc = could not find container \"9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948\": container with ID starting with 9647b8da51235a81a33608d23455cd41668d4effc3f012326464feb92a161948 not found: ID does not exist"
Jul 28 01:59:21.096594 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.096564    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:59:21.100252 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.100226    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-bc47b-5b5f84d749-r4bth"]
Jul 28 01:59:21.145887 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.145851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:21.146084 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.145923    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/77e0c0a6-630f-474c-8d27-bca84a9d28f7-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:59:21.146084 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.145936    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/77e0c0a6-630f-474c-8d27-bca84a9d28f7-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 01:59:21.148376 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.148351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") pod \"switch-graph-704b9-699d45d87-vr6gx\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") " pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:21.353124 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.353083    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:21.484876 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.484813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 01:59:21.487376 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:59:21.487344    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f7fa45_630c_432c_bc84_95bdfeabce3b.slice/crio-9f7c1afadea2a3a55272586dfc6346cb567f11845657b5fd13711f08dd7d02e6 WatchSource:0}: Error finding container 9f7c1afadea2a3a55272586dfc6346cb567f11845657b5fd13711f08dd7d02e6: Status 404 returned error can't find the container with id 9f7c1afadea2a3a55272586dfc6346cb567f11845657b5fd13711f08dd7d02e6
Jul 28 01:59:21.647479 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:21.647398    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" path="/var/lib/kubelet/pods/77e0c0a6-630f-474c-8d27-bca84a9d28f7/volumes"
Jul 28 01:59:22.080481 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:22.080443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" event={"ID":"81f7fa45-630c-432c-bc84-95bdfeabce3b","Type":"ContainerStarted","Data":"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"}
Jul 28 01:59:22.080672 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:22.080501    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" event={"ID":"81f7fa45-630c-432c-bc84-95bdfeabce3b","Type":"ContainerStarted","Data":"9f7c1afadea2a3a55272586dfc6346cb567f11845657b5fd13711f08dd7d02e6"}
Jul 28 01:59:22.080672 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:22.080581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:22.097755 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:22.097696    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podStartSLOduration=2.097678951 podStartE2EDuration="2.097678951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:22.095715089 +0000 UTC m=+2011.001284485" watchObservedRunningTime="2026-07-28 01:59:22.097678951 +0000 UTC m=+2011.003248348"
Jul 28 01:59:28.091179 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:28.091150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 01:59:28.966531 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:28.966486    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 01:59:38.967474 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:38.967446    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 01:59:51.085190 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.085155    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 01:59:51.085673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.085582    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b"
Jul 28 01:59:51.085673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.085599    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b"
Jul 28 01:59:51.085673 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.085673    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="77e0c0a6-630f-474c-8d27-bca84a9d28f7" containerName="splitter-graph-bc47b"
Jul 28 01:59:51.085829 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.085761    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:51.088674 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.088657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.091100 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.091081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7876a-kube-rbac-proxy-sar-config\""
Jul 28 01:59:51.091237 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.091081    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"splitter-graph-7876a-serving-cert\""
Jul 28 01:59:51.096847 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.096825    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 01:59:51.096951 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.096876    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.096999 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.096987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.198211 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.198173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.198211 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.198216    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.198486 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:59:51.198342    2581 secret.go:189] Couldn't get secret kserve-ci-e2e-test/splitter-graph-7876a-serving-cert: secret "splitter-graph-7876a-serving-cert" not found
Jul 28 01:59:51.198486 ip-10-0-134-120 kubenswrapper[2581]: E0728 01:59:51.198401    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls podName:7aa041fb-c1c8-4c82-8cb6-e8478829fb64 nodeName:}" failed. No retries permitted until 2026-07-28 01:59:51.698385482 +0000 UTC m=+2040.603954859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls") pod "splitter-graph-7876a-c666c46dc-zpwsv" (UID: "7aa041fb-c1c8-4c82-8cb6-e8478829fb64") : secret "splitter-graph-7876a-serving-cert" not found
Jul 28 01:59:51.198957 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.198932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.288807 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.288771    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 01:59:51.702357 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.702296    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:51.704644 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.704622    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") pod \"splitter-graph-7876a-c666c46dc-zpwsv\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") " pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:52.000021 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:51.999931    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:52.133941 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:52.133915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 01:59:52.136483 ip-10-0-134-120 kubenswrapper[2581]: W0728 01:59:52.136456    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa041fb_c1c8_4c82_8cb6_e8478829fb64.slice/crio-bb66e04d0b881498049f216d86e2d090be29170ac4e9b414b6d702e965c472d2 WatchSource:0}: Error finding container bb66e04d0b881498049f216d86e2d090be29170ac4e9b414b6d702e965c472d2: Status 404 returned error can't find the container with id bb66e04d0b881498049f216d86e2d090be29170ac4e9b414b6d702e965c472d2
Jul 28 01:59:52.197723 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:52.197692    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" event={"ID":"7aa041fb-c1c8-4c82-8cb6-e8478829fb64","Type":"ContainerStarted","Data":"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"}
Jul 28 01:59:52.197891 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:52.197740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" event={"ID":"7aa041fb-c1c8-4c82-8cb6-e8478829fb64","Type":"ContainerStarted","Data":"bb66e04d0b881498049f216d86e2d090be29170ac4e9b414b6d702e965c472d2"}
Jul 28 01:59:52.197891 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:52.197832    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 01:59:52.212707 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:52.212656    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podStartSLOduration=1.212640867 podStartE2EDuration="1.212640867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 01:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 01:59:52.211273231 +0000 UTC m=+2041.116842623" watchObservedRunningTime="2026-07-28 01:59:52.212640867 +0000 UTC m=+2041.118210262"
Jul 28 01:59:58.207764 ip-10-0-134-120 kubenswrapper[2581]: I0728 01:59:58.207732    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 02:00:51.706789 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:00:51.706756    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:00:51.714943 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:00:51.714909    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:01:03.499483 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:01:03.499448    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:03:30.217349 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:03:30.217252    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:05:51.732715 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:05:51.732681    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:05:51.741560 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:05:51.741532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:06:10.470214 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:06:10.470174    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:08:05.725639 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.725562    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 02:08:05.726214 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.725758    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" containerID="cri-o://7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9" gracePeriod=30
Jul 28 02:08:05.784536 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.784500    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:08:05.863275 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.863238    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 02:08:05.863491 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.863469    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" containerID="cri-o://66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536" gracePeriod=30
Jul 28 02:08:05.863555 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.863504    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kube-rbac-proxy" containerID="cri-o://a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1" gracePeriod=30
Jul 28 02:08:05.889616 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.889583    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:08:05.957736 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:05.957698    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:08:06.195321 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:06.195289    2581 generic.go:356] "Generic (PLEG): container finished" podID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerID="a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1" exitCode=2
Jul 28 02:08:06.195321 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:06.195317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerDied","Data":"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"}
Jul 28 02:08:08.206533 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:08.206490    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:08.961916 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:08.961878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 28 02:08:08.966200 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:08.966178    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 02:08:09.130145 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.130118    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 02:08:09.209759 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.209668    2581 generic.go:356] "Generic (PLEG): container finished" podID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerID="66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536" exitCode=0
Jul 28 02:08:09.209759 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.209726    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"
Jul 28 02:08:09.210299 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.209727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerDied","Data":"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"}
Jul 28 02:08:09.210299 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.209863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7" event={"ID":"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad","Type":"ContainerDied","Data":"42e0e51d6f23935002ba1c5043ffd6caf3ee868d15d14fcc02f2cd0800e2959c"}
Jul 28 02:08:09.210299 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.209892    2581 scope.go:117] "RemoveContainer" containerID="a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"
Jul 28 02:08:09.210557 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.210541    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pbtjj\" (UniqueName: \"kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj\") pod \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") "
Jul 28 02:08:09.210617 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.210586    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") pod \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") "
Jul 28 02:08:09.210684 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.210672    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config\") pod \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\" (UID: \"9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad\") "
Jul 28 02:08:09.211125 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.211097    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" (UID: "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:08:09.212792 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.212760    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" (UID: "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:08:09.212907 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.212833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj" (OuterVolumeSpecName: "kube-api-access-pbtjj") pod "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" (UID: "9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad"). InnerVolumeSpecName "kube-api-access-pbtjj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:08:09.227622 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.227595    2581 scope.go:117] "RemoveContainer" containerID="66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"
Jul 28 02:08:09.235298 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.235279    2581 scope.go:117] "RemoveContainer" containerID="a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"
Jul 28 02:08:09.235616 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:08:09.235595    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1\": container with ID starting with a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1 not found: ID does not exist" containerID="a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"
Jul 28 02:08:09.235683 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.235631    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1"} err="failed to get container status \"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1\": rpc error: code = NotFound desc = could not find container \"a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1\": container with ID starting with a70b877888383d5a0894fac20ac34c7f3d76704e436d177f3e988cea2dd336e1 not found: ID does not exist"
Jul 28 02:08:09.235683 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.235652    2581 scope.go:117] "RemoveContainer" containerID="66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"
Jul 28 02:08:09.235929 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:08:09.235908    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536\": container with ID starting with 66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536 not found: ID does not exist" containerID="66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"
Jul 28 02:08:09.235991 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.235935    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536"} err="failed to get container status \"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536\": rpc error: code = NotFound desc = could not find container \"66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536\": container with ID starting with 66398aa6a2caeb8a1e6219c201c1906007e7d8c74b9588d965b256920d714536 not found: ID does not exist"
Jul 28 02:08:09.311491 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.311455    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.311491 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.311487    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbtjj\" (UniqueName: \"kubernetes.io/projected/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-kube-api-access-pbtjj\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.311491 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.311497    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:08:09.532117 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.532083    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 02:08:09.535547 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.535522    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-7876a-predictor-7b564495c-rrcc7"]
Jul 28 02:08:09.647789 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:09.647757    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" path="/var/lib/kubelet/pods/9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad/volumes"
Jul 28 02:08:13.206837 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:13.206780    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:18.206722 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:18.206682    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:18.207133 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:18.206813    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 02:08:23.206100 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:23.206059    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:28.206873 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:28.206836    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:33.206151 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:33.206111    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:08:35.895998 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:35.895971    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 02:08:36.044756 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.044717    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") pod \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") "
Jul 28 02:08:36.044958 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.044783    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle\") pod \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\" (UID: \"7aa041fb-c1c8-4c82-8cb6-e8478829fb64\") "
Jul 28 02:08:36.045142 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.045122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "7aa041fb-c1c8-4c82-8cb6-e8478829fb64" (UID: "7aa041fb-c1c8-4c82-8cb6-e8478829fb64"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:08:36.046758 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.046735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7aa041fb-c1c8-4c82-8cb6-e8478829fb64" (UID: "7aa041fb-c1c8-4c82-8cb6-e8478829fb64"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:08:36.145482 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.145442    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:08:36.145482 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.145476    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7aa041fb-c1c8-4c82-8cb6-e8478829fb64-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:08:36.316243 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.316150    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerID="7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9" exitCode=0
Jul 28 02:08:36.316243 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.316197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" event={"ID":"7aa041fb-c1c8-4c82-8cb6-e8478829fb64","Type":"ContainerDied","Data":"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"}
Jul 28 02:08:36.316243 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.316198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"
Jul 28 02:08:36.316243 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.316237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv" event={"ID":"7aa041fb-c1c8-4c82-8cb6-e8478829fb64","Type":"ContainerDied","Data":"bb66e04d0b881498049f216d86e2d090be29170ac4e9b414b6d702e965c472d2"}
Jul 28 02:08:36.316571 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.316258    2581 scope.go:117] "RemoveContainer" containerID="7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"
Jul 28 02:08:36.324786 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.324768    2581 scope.go:117] "RemoveContainer" containerID="7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"
Jul 28 02:08:36.325095 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:08:36.325075    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9\": container with ID starting with 7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9 not found: ID does not exist" containerID="7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"
Jul 28 02:08:36.325149 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.325103    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9"} err="failed to get container status \"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9\": rpc error: code = NotFound desc = could not find container \"7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9\": container with ID starting with 7d1525f426bf64a744a203d86c65ab7f10a872a9cf9e977c8107a63e8f197cc9 not found: ID does not exist"
Jul 28 02:08:36.335884 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.335851    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 02:08:36.339638 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:36.339613    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/splitter-graph-7876a-c666c46dc-zpwsv"]
Jul 28 02:08:37.647589 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:08:37.647554    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" path="/var/lib/kubelet/pods/7aa041fb-c1c8-4c82-8cb6-e8478829fb64/volumes"
Jul 28 02:10:51.757318 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:10:51.757236    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:10:51.766672 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:10:51.766647    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:11:24.839958 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:11:24.839922    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:12:38.071825 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:12:38.071727    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:39.502307 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.502220    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 02:15:39.504934 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.502438    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" containerID="cri-o://613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5" gracePeriod=30
Jul 28 02:15:39.560546 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.560507    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:39.670947 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.670908    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 02:15:39.671221 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.671189    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container" containerID="cri-o://53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe" gracePeriod=30
Jul 28 02:15:39.671366 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.671235    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kube-rbac-proxy" containerID="cri-o://86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12" gracePeriod=30
Jul 28 02:15:39.696737 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.696708    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:39.767073 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.766983    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:39.970605 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.970569    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerID="86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12" exitCode=2
Jul 28 02:15:39.970779 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:39.970650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerDied","Data":"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"}
Jul 28 02:15:40.808427 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808390    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wcz8d/must-gather-fdrfv"]
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808737    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808749    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808761    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kube-rbac-proxy"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kube-rbac-proxy"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808776    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808782    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container"
Jul 28 02:15:40.808841 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808844    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kube-rbac-proxy"
Jul 28 02:15:40.809072 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808854    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aa041fb-c1c8-4c82-8cb6-e8478829fb64" containerName="splitter-graph-7876a"
Jul 28 02:15:40.809072 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.808861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fef3c61-9e7b-4a3c-9c37-f15a06fcf3ad" containerName="kserve-container"
Jul 28 02:15:40.811955 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.811935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:40.814383 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.814360    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wcz8d\"/\"kube-root-ca.crt\""
Jul 28 02:15:40.814955 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.814930    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wcz8d\"/\"default-dockercfg-bbs7z\""
Jul 28 02:15:40.814955 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.814934    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wcz8d\"/\"openshift-service-ca.crt\""
Jul 28 02:15:40.823540 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.823509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wcz8d/must-gather-fdrfv"]
Jul 28 02:15:40.827135 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.827107    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:40.927395 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.927359    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:15:40.966663 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.966623    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6h46\" (UniqueName: \"kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:40.966834 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:40.966733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.068213 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.068111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6h46\" (UniqueName: \"kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.068213 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.068173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.068501 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.068481    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.077415 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.077380    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6h46\" (UniqueName: \"kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46\") pod \"must-gather-fdrfv\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") " pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.130931 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.130891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:15:41.270214 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.270169    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wcz8d/must-gather-fdrfv"]
Jul 28 02:15:41.273387 ip-10-0-134-120 kubenswrapper[2581]: W0728 02:15:41.273359    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21a7b5c7_5a4e_4ee7_b3d8_3f0387470369.slice/crio-434ba77e04c65fb1490375c6549e1c81bbd9c3030658ae340fdee9ffc0a7aa5b WatchSource:0}: Error finding container 434ba77e04c65fb1490375c6549e1c81bbd9c3030658ae340fdee9ffc0a7aa5b: Status 404 returned error can't find the container with id 434ba77e04c65fb1490375c6549e1c81bbd9c3030658ae340fdee9ffc0a7aa5b
Jul 28 02:15:41.275043 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.275027    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 02:15:41.986696 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:41.986660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" event={"ID":"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369","Type":"ContainerStarted","Data":"434ba77e04c65fb1490375c6549e1c81bbd9c3030658ae340fdee9ffc0a7aa5b"}
Jul 28 02:15:42.778975 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:42.778930    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 02:15:43.091007 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.090905    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:15:43.358022 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.357946    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 02:15:43.492471 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.492438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wxsfv\" (UniqueName: \"kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv\") pod \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") "
Jul 28 02:15:43.492675 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.492518    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") pod \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") "
Jul 28 02:15:43.492675 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.492608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config\") pod \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\" (UID: \"8b088f90-c19d-4e15-acf4-210df4ecf4b4\") "
Jul 28 02:15:43.493202 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.493172    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8b088f90-c19d-4e15-acf4-210df4ecf4b4" (UID: "8b088f90-c19d-4e15-acf4-210df4ecf4b4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:15:43.495290 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.495259    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8b088f90-c19d-4e15-acf4-210df4ecf4b4" (UID: "8b088f90-c19d-4e15-acf4-210df4ecf4b4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:15:43.495402 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.495346    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv" (OuterVolumeSpecName: "kube-api-access-wxsfv") pod "8b088f90-c19d-4e15-acf4-210df4ecf4b4" (UID: "8b088f90-c19d-4e15-acf4-210df4ecf4b4"). InnerVolumeSpecName "kube-api-access-wxsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:15:43.594070 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.594029    2581 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.594070 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.594064    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxsfv\" (UniqueName: \"kubernetes.io/projected/8b088f90-c19d-4e15-acf4-210df4ecf4b4-kube-api-access-wxsfv\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.594070 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.594080    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8b088f90-c19d-4e15-acf4-210df4ecf4b4-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:15:43.997912 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.997823    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerID="53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe" exitCode=0
Jul 28 02:15:43.998076 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.997928    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerDied","Data":"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"}
Jul 28 02:15:43.998076 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.997955    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"
Jul 28 02:15:43.998076 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.997990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm" event={"ID":"8b088f90-c19d-4e15-acf4-210df4ecf4b4","Type":"ContainerDied","Data":"54e31fc2e298ce60e92f983b35ebd840ca7203410bfc110485625b5412e68391"}
Jul 28 02:15:43.998076 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:43.998020    2581 scope.go:117] "RemoveContainer" containerID="86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"
Jul 28 02:15:44.016288 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:44.016254    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 02:15:44.023007 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:44.022969    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-704b9-predictor-5774cf759-k29gm"]
Jul 28 02:15:45.650341 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:45.650303    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" path="/var/lib/kubelet/pods/8b088f90-c19d-4e15-acf4-210df4ecf4b4/volumes"
Jul 28 02:15:46.553065 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:46.552912    2581 scope.go:117] "RemoveContainer" containerID="53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"
Jul 28 02:15:47.501751 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:47.501730    2581 scope.go:117] "RemoveContainer" containerID="86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"
Jul 28 02:15:47.502171 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:15:47.502113    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12\": container with ID starting with 86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12 not found: ID does not exist" containerID="86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"
Jul 28 02:15:47.502171 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:47.502145    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12"} err="failed to get container status \"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12\": rpc error: code = NotFound desc = could not find container \"86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12\": container with ID starting with 86e11979bfcffe8c29ac99a1682a582bbe2c77e7b8fc98ae6ea788e2d3395e12 not found: ID does not exist"
Jul 28 02:15:47.502257 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:47.502172    2581 scope.go:117] "RemoveContainer" containerID="53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"
Jul 28 02:15:47.502477 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:15:47.502458    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe\": container with ID starting with 53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe not found: ID does not exist" containerID="53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"
Jul 28 02:15:47.502522 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:47.502484    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe"} err="failed to get container status \"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe\": rpc error: code = NotFound desc = could not find container \"53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe\": container with ID starting with 53b2aec545e34c614df8f0e2d4cc362447abe28e4d60e3e4a0f12e6cd6dd20fe not found: ID does not exist"
Jul 28 02:15:48.022273 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:48.022230    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" event={"ID":"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369","Type":"ContainerStarted","Data":"432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e"}
Jul 28 02:15:48.022421 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:48.022285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" event={"ID":"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369","Type":"ContainerStarted","Data":"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"}
Jul 28 02:15:48.045481 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:48.045355    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" podStartSLOduration=1.749486718 podStartE2EDuration="8.045338221s" totalImagesPullingTime="6.295851503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:15:48.044316581 +0000 UTC m=+2996.949885981" watchObservedRunningTime="2026-07-28 02:15:48.045338221 +0000 UTC m=+2996.950907619"
Jul 28 02:15:48.089774 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:48.089735    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:15:51.796441 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:51.796413    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:15:51.802391 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:51.802365    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:15:53.089501 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:53.089455    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:15:53.090005 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:53.089591    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 02:15:57.085688 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:57.085641    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:15:57.916029 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:57.915996    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:15:58.090137 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:58.090092    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:15:58.753683 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:58.753652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:15:59.560000 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:15:59.559968    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:00.351564 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:00.351532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:01.169067 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:01.169035    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:01.982489 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:01.982460    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:02.805016 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:02.804985    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:03.089734 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:03.089632    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:16:03.619787 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:03.619759    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:04.420740 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:04.420699    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:05.249425 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:05.249359    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:06.083253 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:06.083221    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:06.913880 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:06.913842    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:07.781725 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:07.781694    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_switch-graph-704b9-699d45d87-vr6gx_81f7fa45-630c-432c-bc84-95bdfeabce3b/switch-graph-704b9/0.log"
Jul 28 02:16:08.088977 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:08.088878    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 02:16:09.661181 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.661153    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 02:16:09.712125 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.712089    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") pod \"81f7fa45-630c-432c-bc84-95bdfeabce3b\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") "
Jul 28 02:16:09.712308 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.712147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle\") pod \"81f7fa45-630c-432c-bc84-95bdfeabce3b\" (UID: \"81f7fa45-630c-432c-bc84-95bdfeabce3b\") "
Jul 28 02:16:09.712543 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.712510    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle" (OuterVolumeSpecName: "openshift-service-ca-bundle") pod "81f7fa45-630c-432c-bc84-95bdfeabce3b" (UID: "81f7fa45-630c-432c-bc84-95bdfeabce3b"). InnerVolumeSpecName "openshift-service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 02:16:09.714227 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.714198    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "81f7fa45-630c-432c-bc84-95bdfeabce3b" (UID: "81f7fa45-630c-432c-bc84-95bdfeabce3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 02:16:09.812977 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.812884    2581 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/81f7fa45-630c-432c-bc84-95bdfeabce3b-proxy-tls\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:16:09.812977 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:09.812917    2581 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81f7fa45-630c-432c-bc84-95bdfeabce3b-openshift-service-ca-bundle\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:16:10.121250 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.121142    2581 generic.go:356] "Generic (PLEG): container finished" podID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerID="613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5" exitCode=0
Jul 28 02:16:10.121250 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.121189    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"
Jul 28 02:16:10.121250 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.121237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" event={"ID":"81f7fa45-630c-432c-bc84-95bdfeabce3b","Type":"ContainerDied","Data":"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"}
Jul 28 02:16:10.121533 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.121299    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx" event={"ID":"81f7fa45-630c-432c-bc84-95bdfeabce3b","Type":"ContainerDied","Data":"9f7c1afadea2a3a55272586dfc6346cb567f11845657b5fd13711f08dd7d02e6"}
Jul 28 02:16:10.121533 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.121331    2581 scope.go:117] "RemoveContainer" containerID="613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"
Jul 28 02:16:10.123478 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.123448    2581 generic.go:356] "Generic (PLEG): container finished" podID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerID="5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627" exitCode=0
Jul 28 02:16:10.123586 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.123480    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" event={"ID":"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369","Type":"ContainerDied","Data":"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"}
Jul 28 02:16:10.123918 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.123899    2581 scope.go:117] "RemoveContainer" containerID="5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"
Jul 28 02:16:10.130274 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.130257    2581 scope.go:117] "RemoveContainer" containerID="613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"
Jul 28 02:16:10.130591 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:10.130569    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5\": container with ID starting with 613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5 not found: ID does not exist" containerID="613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"
Jul 28 02:16:10.130670 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.130604    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5"} err="failed to get container status \"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5\": rpc error: code = NotFound desc = could not find container \"613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5\": container with ID starting with 613931241d52a6d38572c0fd1bd3abd0a66ecd0b5e64ff0b8725de8e06d761a5 not found: ID does not exist"
Jul 28 02:16:10.150705 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.150667    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 02:16:10.152437 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.152411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/switch-graph-704b9-699d45d87-vr6gx"]
Jul 28 02:16:10.282912 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.282880    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wcz8d_must-gather-fdrfv_21a7b5c7-5a4e-4ee7-b3d8-3f0387470369/gather/0.log"
Jul 28 02:16:10.731755 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.731665    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:10.852947 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.852908    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8l4r/must-gather-pxvm5"]
Jul 28 02:16:10.853245 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853232    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9"
Jul 28 02:16:10.853290 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853247    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9"
Jul 28 02:16:10.853290 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container"
Jul 28 02:16:10.853290 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853270    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container"
Jul 28 02:16:10.853290 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853283    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kube-rbac-proxy"
Jul 28 02:16:10.853414 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853293    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kube-rbac-proxy"
Jul 28 02:16:10.853414 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853340    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kserve-container"
Jul 28 02:16:10.853414 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853348    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" containerName="switch-graph-704b9"
Jul 28 02:16:10.853414 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.853358    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b088f90-c19d-4e15-acf4-210df4ecf4b4" containerName="kube-rbac-proxy"
Jul 28 02:16:10.856484 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.856465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:10.859075 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.859045    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8l4r\"/\"openshift-service-ca.crt\""
Jul 28 02:16:10.859230 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.859128    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p8l4r\"/\"kube-root-ca.crt\""
Jul 28 02:16:10.859506 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.859491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p8l4r\"/\"default-dockercfg-gpg4w\""
Jul 28 02:16:10.865339 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.865310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/must-gather-pxvm5"]
Jul 28 02:16:10.879862 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.879829    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:10.923322 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.923278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b9dd\" (UniqueName: \"kubernetes.io/projected/6df7f1ab-6e64-4408-8151-2549787e48cd-kube-api-access-4b9dd\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:10.923510 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.923368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df7f1ab-6e64-4408-8151-2549787e48cd-must-gather-output\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:10.980073 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:10.980043    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:11.024443 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.024408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df7f1ab-6e64-4408-8151-2549787e48cd-must-gather-output\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:11.024585 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.024477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b9dd\" (UniqueName: \"kubernetes.io/projected/6df7f1ab-6e64-4408-8151-2549787e48cd-kube-api-access-4b9dd\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:11.024757 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.024738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df7f1ab-6e64-4408-8151-2549787e48cd-must-gather-output\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:11.032499 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.032475    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b9dd\" (UniqueName: \"kubernetes.io/projected/6df7f1ab-6e64-4408-8151-2549787e48cd-kube-api-access-4b9dd\") pod \"must-gather-pxvm5\" (UID: \"6df7f1ab-6e64-4408-8151-2549787e48cd\") " pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:11.166855 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.166818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/must-gather-pxvm5"
Jul 28 02:16:11.304160 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.304084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/must-gather-pxvm5"]
Jul 28 02:16:11.307017 ip-10-0-134-120 kubenswrapper[2581]: W0728 02:16:11.306988    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df7f1ab_6e64_4408_8151_2549787e48cd.slice/crio-d3a3927d6b7ac1caeb1e3bdbb53c108e080e42833658342423416096d3a26503 WatchSource:0}: Error finding container d3a3927d6b7ac1caeb1e3bdbb53c108e080e42833658342423416096d3a26503: Status 404 returned error can't find the container with id d3a3927d6b7ac1caeb1e3bdbb53c108e080e42833658342423416096d3a26503
Jul 28 02:16:11.647412 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:11.647331    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f7fa45-630c-432c-bc84-95bdfeabce3b" path="/var/lib/kubelet/pods/81f7fa45-630c-432c-bc84-95bdfeabce3b/volumes"
Jul 28 02:16:12.135998 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:12.135919    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/must-gather-pxvm5" event={"ID":"6df7f1ab-6e64-4408-8151-2549787e48cd","Type":"ContainerStarted","Data":"ba332fe36225d30ecbf14d9b15d3c9d0017abd81057501f313cfbcf6d5b54452"}
Jul 28 02:16:12.135998 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:12.135973    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/must-gather-pxvm5" event={"ID":"6df7f1ab-6e64-4408-8151-2549787e48cd","Type":"ContainerStarted","Data":"d3a3927d6b7ac1caeb1e3bdbb53c108e080e42833658342423416096d3a26503"}
Jul 28 02:16:13.143161 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:13.143097    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/must-gather-pxvm5" event={"ID":"6df7f1ab-6e64-4408-8151-2549787e48cd","Type":"ContainerStarted","Data":"adea848879ce996246992476aafe03a9ba15fcb4e585372ea5685f23c439ad9c"}
Jul 28 02:16:13.159098 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:13.159042    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8l4r/must-gather-pxvm5" podStartSLOduration=2.501566268 podStartE2EDuration="3.158986385s" totalImagesPullingTime="657.420117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 02:16:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:16:13.158173731 +0000 UTC m=+3022.063743142" watchObservedRunningTime="2026-07-28 02:16:13.158986385 +0000 UTC m=+3022.064592883"
Jul 28 02:16:13.413069 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:13.412935    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hsjxk_9d8f5494-05fc-48a7-a8c2-cbfbf30134fb/global-pull-secret-syncer/0.log"
Jul 28 02:16:13.508491 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:13.508461    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log"
Jul 28 02:16:13.622623 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:13.622595    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-120.ec2.internal_3f6d61b851655e512466d358f996bf98/haproxy/0.log"
Jul 28 02:16:15.715437 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:15.715385    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-wcz8d/must-gather-fdrfv"]
Jul 28 02:16:15.715976 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:15.715925    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-wcz8d/must-gather-fdrfv" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="copy" containerID="cri-o://432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e" gracePeriod=2
Jul 28 02:16:15.718466 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:15.718396    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fdrfv\" is forbidden: User \"system:node:ip-10-0-134-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wcz8d\": no relationship found between node 'ip-10-0-134-120.ec2.internal' and this object" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:15.719976 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:15.719947    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-wcz8d/must-gather-fdrfv"]
Jul 28 02:16:16.092563 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.091107    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wcz8d_must-gather-fdrfv_21a7b5c7-5a4e-4ee7-b3d8-3f0387470369/copy/0.log"
Jul 28 02:16:16.094866 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.093426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:16.098857 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:16.098785    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fdrfv\" is forbidden: User \"system:node:ip-10-0-134-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wcz8d\": no relationship found between node 'ip-10-0-134-120.ec2.internal' and this object" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:16.162891 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.162784    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wcz8d_must-gather-fdrfv_21a7b5c7-5a4e-4ee7-b3d8-3f0387470369/copy/0.log"
Jul 28 02:16:16.164333 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.163514    2581 generic.go:356] "Generic (PLEG): container finished" podID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerID="432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e" exitCode=143
Jul 28 02:16:16.164333 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.163602    2581 scope.go:117] "RemoveContainer" containerID="432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e"
Jul 28 02:16:16.164333 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.163758    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:16.166319 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:16.166243    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fdrfv\" is forbidden: User \"system:node:ip-10-0-134-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wcz8d\": no relationship found between node 'ip-10-0-134-120.ec2.internal' and this object" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:16.179286 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.178708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6h46\" (UniqueName: \"kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46\") pod \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") "
Jul 28 02:16:16.179286 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.178754    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output\") pod \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\" (UID: \"21a7b5c7-5a4e-4ee7-b3d8-3f0387470369\") "
Jul 28 02:16:16.181892 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.181501    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" (UID: "21a7b5c7-5a4e-4ee7-b3d8-3f0387470369"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 02:16:16.186744 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.185739    2581 scope.go:117] "RemoveContainer" containerID="5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"
Jul 28 02:16:16.193220 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.193181    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46" (OuterVolumeSpecName: "kube-api-access-z6h46") pod "21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" (UID: "21a7b5c7-5a4e-4ee7-b3d8-3f0387470369"). InnerVolumeSpecName "kube-api-access-z6h46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 02:16:16.215132 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.214988    2581 scope.go:117] "RemoveContainer" containerID="432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e"
Jul 28 02:16:16.215746 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:16.215604    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e\": container with ID starting with 432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e not found: ID does not exist" containerID="432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e"
Jul 28 02:16:16.215746 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.215645    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e"} err="failed to get container status \"432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e\": rpc error: code = NotFound desc = could not find container \"432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e\": container with ID starting with 432d9a1eec7873bcb5db2ae4274702e4c162cce414962a097e976927193af31e not found: ID does not exist"
Jul 28 02:16:16.215746 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.215667    2581 scope.go:117] "RemoveContainer" containerID="5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"
Jul 28 02:16:16.216416 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:16.216340    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627\": container with ID starting with 5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627 not found: ID does not exist" containerID="5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"
Jul 28 02:16:16.216416 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.216366    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627"} err="failed to get container status \"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627\": rpc error: code = NotFound desc = could not find container \"5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627\": container with ID starting with 5429139301db8e92dda5fc2ca4088c2e57287602aa73de8da43fb1d34788d627 not found: ID does not exist"
Jul 28 02:16:16.279990 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.279913    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6h46\" (UniqueName: \"kubernetes.io/projected/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-kube-api-access-z6h46\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:16:16.279990 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.279950    2581 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369-must-gather-output\") on node \"ip-10-0-134-120.ec2.internal\" DevicePath \"\""
Jul 28 02:16:16.347688 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.347572    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:16.448487 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:16.448347    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:16.480542 ip-10-0-134-120 kubenswrapper[2581]: E0728 02:16:16.480064    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-fdrfv\" is forbidden: User \"system:node:ip-10-0-134-120.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wcz8d\": no relationship found between node 'ip-10-0-134-120.ec2.internal' and this object" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" pod="openshift-must-gather-wcz8d/must-gather-fdrfv"
Jul 28 02:16:17.581224 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.581175    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lmxt4_c0dd6710-e852-46bb-991c-cac90d69c12d/kube-state-metrics/0.log"
Jul 28 02:16:17.610176 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.610110    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lmxt4_c0dd6710-e852-46bb-991c-cac90d69c12d/kube-rbac-proxy-main/0.log"
Jul 28 02:16:17.644084 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.644054    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-lmxt4_c0dd6710-e852-46bb-991c-cac90d69c12d/kube-rbac-proxy-self/0.log"
Jul 28 02:16:17.651111 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.651076    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" path="/var/lib/kubelet/pods/21a7b5c7-5a4e-4ee7-b3d8-3f0387470369/volumes"
Jul 28 02:16:17.735383 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.735352    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-vbj2j_a780f93f-13ea-4bd5-b370-64c18d4e5c60/monitoring-plugin/0.log"
Jul 28 02:16:17.771931 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.771901    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qq55_f1c85108-418d-4c11-8762-8231d4f84826/node-exporter/0.log"
Jul 28 02:16:17.802008 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.801969    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qq55_f1c85108-418d-4c11-8762-8231d4f84826/kube-rbac-proxy/0.log"
Jul 28 02:16:17.833223 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:17.833151    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2qq55_f1c85108-418d-4c11-8762-8231d4f84826/init-textfile/0.log"
Jul 28 02:16:18.134080 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.133991    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jxxst_3ed4e918-20c2-4293-acb8-d32ada84f2eb/kube-rbac-proxy-main/0.log"
Jul 28 02:16:18.163170 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.163135    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jxxst_3ed4e918-20c2-4293-acb8-d32ada84f2eb/kube-rbac-proxy-self/0.log"
Jul 28 02:16:18.192620 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.192583    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-jxxst_3ed4e918-20c2-4293-acb8-d32ada84f2eb/openshift-state-metrics/0.log"
Jul 28 02:16:18.246070 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.246031    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/prometheus/0.log"
Jul 28 02:16:18.273571 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.273539    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/config-reloader/0.log"
Jul 28 02:16:18.301727 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.301700    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/thanos-sidecar/0.log"
Jul 28 02:16:18.334696 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.334638    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/kube-rbac-proxy-web/0.log"
Jul 28 02:16:18.362278 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.362243    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/kube-rbac-proxy/0.log"
Jul 28 02:16:18.395575 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.395482    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/kube-rbac-proxy-thanos/0.log"
Jul 28 02:16:18.427102 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.427071    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_4cecabe8-0238-4a97-9ca7-ac890af12940/init-config-reloader/0.log"
Jul 28 02:16:18.463625 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.463593    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ptrd2_05176fac-afd9-4b0a-baca-04a1fdf0a260/prometheus-operator/0.log"
Jul 28 02:16:18.488180 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.488146    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-ptrd2_05176fac-afd9-4b0a-baca-04a1fdf0a260/kube-rbac-proxy/0.log"
Jul 28 02:16:18.559169 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.559132    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc868b465-cpngh_2ac8110e-26ae-412f-8188-bd530bf3f0d4/telemeter-client/0.log"
Jul 28 02:16:18.586189 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.586154    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc868b465-cpngh_2ac8110e-26ae-412f-8188-bd530bf3f0d4/reload/0.log"
Jul 28 02:16:18.612987 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:18.612955    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7fc868b465-cpngh_2ac8110e-26ae-412f-8188-bd530bf3f0d4/kube-rbac-proxy/0.log"
Jul 28 02:16:21.477768 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:21.477736    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-hw9f2_886e9d54-5d8a-4d0d-b1af-bd2963f51ac4/download-server/0.log"
Jul 28 02:16:22.559724 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.559674    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"]
Jul 28 02:16:22.560242 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560194    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="gather"
Jul 28 02:16:22.560242 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560215    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="gather"
Jul 28 02:16:22.560242 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560240    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="copy"
Jul 28 02:16:22.560429 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560249    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="copy"
Jul 28 02:16:22.560429 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560386    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="copy"
Jul 28 02:16:22.560429 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.560407    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="21a7b5c7-5a4e-4ee7-b3d8-3f0387470369" containerName="gather"
Jul 28 02:16:22.564752 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.564718    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.572337 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.572308    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"]
Jul 28 02:16:22.646592 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.646552    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-lib-modules\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.646960 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.646935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jwx4\" (UniqueName: \"kubernetes.io/projected/506bd463-86a2-41e5-af32-7e41bd3fc3bd-kube-api-access-9jwx4\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.647190 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.647174    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-proc\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.647335 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.647321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-podres\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.647489 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.647475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-sys\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748545 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748505    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-lib-modules\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748545 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jwx4\" (UniqueName: \"kubernetes.io/projected/506bd463-86a2-41e5-af32-7e41bd3fc3bd-kube-api-access-9jwx4\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748587    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-proc\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748603    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-podres\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748628    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-sys\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748686    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-proc\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748717    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-sys\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.748776 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748772    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-podres\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.749014 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.748791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/506bd463-86a2-41e5-af32-7e41bd3fc3bd-lib-modules\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.756668 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.756643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jwx4\" (UniqueName: \"kubernetes.io/projected/506bd463-86a2-41e5-af32-7e41bd3fc3bd-kube-api-access-9jwx4\") pod \"perf-node-gather-daemonset-ckk6l\" (UID: \"506bd463-86a2-41e5-af32-7e41bd3fc3bd\") " pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:22.879906 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:22.879793    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:23.038440 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.038407    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"]
Jul 28 02:16:23.044032 ip-10-0-134-120 kubenswrapper[2581]: W0728 02:16:23.044002    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod506bd463_86a2_41e5_af32_7e41bd3fc3bd.slice/crio-e71044dc31ce701bfbb256d887d7477e74854af5b753a06bd61b1d9d0ccbd210 WatchSource:0}: Error finding container e71044dc31ce701bfbb256d887d7477e74854af5b753a06bd61b1d9d0ccbd210: Status 404 returned error can't find the container with id e71044dc31ce701bfbb256d887d7477e74854af5b753a06bd61b1d9d0ccbd210
Jul 28 02:16:23.069464 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.069440    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q9h5v_0806ff13-11a2-41c7-8c0b-84f6788b459a/dns/0.log"
Jul 28 02:16:23.091930 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.091904    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q9h5v_0806ff13-11a2-41c7-8c0b-84f6788b459a/kube-rbac-proxy/0.log"
Jul 28 02:16:23.148340 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.148316    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-lt4pp_25d03f91-4264-43d2-85ae-ca0ffef006a7/dns-node-resolver/0.log"
Jul 28 02:16:23.208474 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.207597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l" event={"ID":"506bd463-86a2-41e5-af32-7e41bd3fc3bd","Type":"ContainerStarted","Data":"730b1fee0ebffe1c9fd81be8ed9efbfafecda30b9255f700cd06ccd595fcc9f1"}
Jul 28 02:16:23.208474 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.207660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l" event={"ID":"506bd463-86a2-41e5-af32-7e41bd3fc3bd","Type":"ContainerStarted","Data":"e71044dc31ce701bfbb256d887d7477e74854af5b753a06bd61b1d9d0ccbd210"}
Jul 28 02:16:23.208474 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.208408    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:23.224112 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.224040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l" podStartSLOduration=1.224017499 podStartE2EDuration="1.224017499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 02:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 02:16:23.223144518 +0000 UTC m=+3032.128713909" watchObservedRunningTime="2026-07-28 02:16:23.224017499 +0000 UTC m=+3032.129586898"
Jul 28 02:16:23.734195 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:23.734153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wnrqq_7878005c-1573-4937-8d3a-98bf70391b9e/node-ca/0.log"
Jul 28 02:16:24.915060 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:24.915024    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kmhz4_962dc654-599f-420d-979a-95ce484e7b2a/serve-healthcheck-canary/0.log"
Jul 28 02:16:25.470764 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:25.470723    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pld79_34b5cb41-e7b0-4e5d-9962-7ee44bba7914/kube-rbac-proxy/0.log"
Jul 28 02:16:25.492383 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:25.492358    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pld79_34b5cb41-e7b0-4e5d-9962-7ee44bba7914/exporter/0.log"
Jul 28 02:16:25.515036 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:25.515008    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-pld79_34b5cb41-e7b0-4e5d-9962-7ee44bba7914/extractor/0.log"
Jul 28 02:16:27.838304 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:27.838274    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-tfpx5_c19b2a1a-a1ce-4ab6-af4b-0f4e9d15b75a/server/0.log"
Jul 28 02:16:27.945652 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:27.945578    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-hq65f_0b83d844-05b5-4576-9a9e-71c438e011c5/manager/0.log"
Jul 28 02:16:27.977965 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:27.977928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8nncc_3c4eb5b0-a3b9-4463-9099-4c5aa33853e6/s3-init/0.log"
Jul 28 02:16:27.995564 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:27.995539    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-8nncc_3c4eb5b0-a3b9-4463-9099-4c5aa33853e6/download-hf-model/0.log"
Jul 28 02:16:30.228830 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:30.228774    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p8l4r/perf-node-gather-daemonset-ckk6l"
Jul 28 02:16:30.720944 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:30.720908    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cq2hp_831129bd-28d1-453f-bd7c-732dbaf3ccf4/konnectivity-agent/0.log" limit=1024
Jul 28 02:16:33.589911 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.589883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/kube-multus-additional-cni-plugins/0.log"
Jul 28 02:16:33.612886 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.612852    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/egress-router-binary-copy/0.log"
Jul 28 02:16:33.636273 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.636230    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/cni-plugins/0.log"
Jul 28 02:16:33.659072 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.659045    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/bond-cni-plugin/0.log"
Jul 28 02:16:33.682879 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.682850    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/routeoverride-cni/0.log"
Jul 28 02:16:33.710287 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.710263    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/whereabouts-cni-bincopy/0.log"
Jul 28 02:16:33.733082 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.733055    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-mvbd5_119d49ad-c20e-42b4-ba59-4fead814dd6e/whereabouts-cni/0.log"
Jul 28 02:16:33.971826 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:33.971708    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vc4kz_582e892b-4304-42c6-a258-71cca1770646/kube-multus/0.log"
Jul 28 02:16:34.028251 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:34.028217    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tr4zt_329df437-fb66-4775-a16e-94b06ef0cf80/network-metrics-daemon/0.log"
Jul 28 02:16:34.050698 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:34.050669    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tr4zt_329df437-fb66-4775-a16e-94b06ef0cf80/kube-rbac-proxy/0.log"
Jul 28 02:16:35.338952 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.338918    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-controller/0.log"
Jul 28 02:16:35.358839 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.358781    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/1.log"
Jul 28 02:16:35.358992 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.358851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovn-acl-logging/0.log"
Jul 28 02:16:35.377036 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.376998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/kube-rbac-proxy-node/0.log"
Jul 28 02:16:35.400160 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.400130    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 02:16:35.424641 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.424618    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/northd/0.log"
Jul 28 02:16:35.448223 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.448198    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/nbdb/0.log"
Jul 28 02:16:35.470892 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.470862    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/sbdb/0.log"
Jul 28 02:16:35.520177 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:35.520138    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-zkwqs_c18fe994-d6a9-4deb-a295-243b7d78334d/ovnkube-controller/0.log"
Jul 28 02:16:36.582306 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:36.582273    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-g79hr_c8bd1caa-d681-4e7c-9361-e5fba9a8aaa7/network-check-target-container/0.log"
Jul 28 02:16:37.502782 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:37.502752    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-g4rcm_68877fd4-0c06-4342-85ea-efa14703ff7e/iptables-alerter/0.log"
Jul 28 02:16:38.174129 ip-10-0-134-120 kubenswrapper[2581]: I0728 02:16:38.174100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-h6hhd_6aa0e6e3-a03d-43c8-9bef-090e34f885d4/tuned/0.log"