Aug 04 17:58:13.683398 ip-10-0-140-15 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Aug 04 17:58:13.683409 ip-10-0-140-15 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Aug 04 17:58:13.683415 ip-10-0-140-15 systemd[1]: kubelet.service: Failed with result 'resources'.
Aug 04 17:58:13.683674 ip-10-0-140-15 systemd[1]: Failed to start Kubernetes Kubelet.
Aug 04 17:58:23.856713 ip-10-0-140-15 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Aug 04 17:58:23.856753 ip-10-0-140-15 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot cafd48bed3254d2c83bf9760ea921800 --
Aug 04 18:00:21.039617 ip-10-0-140-15 systemd[1]: Starting Kubernetes Kubelet...
Aug 04 18:00:21.388926 ip-10-0-140-15 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:21.388926 ip-10-0-140-15 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Aug 04 18:00:21.388926 ip-10-0-140-15 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:21.388926 ip-10-0-140-15 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Aug 04 18:00:21.388926 ip-10-0-140-15 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Aug 04 18:00:21.389846 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.389733    2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Aug 04 18:00:21.393525 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393511    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.393525 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393525    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393530    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393534    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393537    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393540    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393542    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393545    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393548    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393550    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393553    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393555    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393558    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393561    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393563    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393566    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393568    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393570    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393573    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393576    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393578    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.393589 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393580    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393583    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393586    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393588    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393591    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393594    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393597    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393606    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393609    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393612    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393614    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393617    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393619    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393622    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393624    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393627    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393629    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393631    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393634    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393636    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.394063 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393639    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393641    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393643    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393646    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393648    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393651    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393654    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393656    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393659    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393661    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393664    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393673    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393677    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393680    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393683    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393686    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393689    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393691    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393694    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.394568 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393698    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393700    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393703    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393706    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393708    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393711    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393714    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393716    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393719    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393721    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393724    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393727    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393730    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393733    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393735    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393737    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393740    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393743    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393745    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393748    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.395025 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393751    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393753    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393757    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393760    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393763    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393765    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393768    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393771    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393774    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393776    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393779    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393781    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393784    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393786    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393790    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393793    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393796    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393799    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.393802    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.395613 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394556    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394563    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394567    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394570    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394573    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394577    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394580    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394582    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394585    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394587    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394590    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394593    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394595    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394598    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394601    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394604    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394606    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394609    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394611    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394614    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.396107 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394617    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394620    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394624    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394629    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394632    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394635    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394638    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394640    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394643    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394646    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394649    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394652    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394654    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394657    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394660    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394663    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394666    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394670    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394672    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394675    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.396604 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394678    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394681    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394683    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394686    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394689    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394692    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394694    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394697    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394700    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394703    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394705    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394708    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394711    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394713    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394716    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394718    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394721    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394723    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394726    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394728    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.397097 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394731    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394733    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394737    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394741    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394743    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394746    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394748    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394751    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394755    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394757    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394760    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394762    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394765    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394768    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394771    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394773    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394776    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394778    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394781    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.397598 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394783    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394786    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394789    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394791    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394794    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394796    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394799    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394802    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394804    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394807    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394809    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394812    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394814    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394816    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394819    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394821    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394824    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394826    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394829    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.398078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.394832    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395241    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395248    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395253    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395256    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395260    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395263    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395265    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395268    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395271    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395273    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395276    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395278    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395281    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395284    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395287    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395290    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395292    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395295    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395298    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.398558 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395300    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395303    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395306    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395308    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395311    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395314    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395316    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395319    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395321    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395324    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395326    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395329    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395332    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395334    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395337    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395340    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395342    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395346    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395349    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395351    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.399040 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395354    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395356    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395358    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395361    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395363    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395366    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395368    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395371    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395374    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395379    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395381    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395384    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395386    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395389    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395391    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395394    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395396    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395399    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395401    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.399534 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395404    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395407    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395411    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395413    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395416    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395419    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395422    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395424    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395427    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395429    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395432    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395434    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395437    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395439    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395441    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395446    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395449    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395452    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395455    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.400010 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395457    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395460    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395463    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395467    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395469    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395472    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395474    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395477    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395480    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395482    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395485    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395487    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395490    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395492    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395494    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395497    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395500    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395502    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395506    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395509    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.400521 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395511    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.395514    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395575    2554 flags.go:64] FLAG: --address="0.0.0.0"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395581    2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395588    2554 flags.go:64] FLAG: --anonymous-auth="true"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395592    2554 flags.go:64] FLAG: --application-metrics-count-limit="100"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395596    2554 flags.go:64] FLAG: --authentication-token-webhook="false"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395599    2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395603    2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395610    2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395613    2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395617    2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395620    2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395624    2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395627    2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395631    2554 flags.go:64] FLAG: --cgroup-root=""
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395634    2554 flags.go:64] FLAG: --cgroups-per-qos="true"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395637    2554 flags.go:64] FLAG: --client-ca-file=""
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395639    2554 flags.go:64] FLAG: --cloud-provider="external"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395642    2554 flags.go:64] FLAG: --cluster-dns="[]"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395646    2554 flags.go:64] FLAG: --cluster-domain=""
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395649    2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395652    2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Aug 04 18:00:21.401053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395654    2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395658    2554 flags.go:64] FLAG: --container-log-max-files="5"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395661    2554 flags.go:64] FLAG: --container-log-max-size="10Mi"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395664    2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395667    2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395670    2554 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395673    2554 flags.go:64] FLAG: --contention-profiling="false"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395677    2554 flags.go:64] FLAG: --cpu-cfs-quota="true"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395680    2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395683    2554 flags.go:64] FLAG: --cpu-manager-policy="none"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395685    2554 flags.go:64] FLAG: --cpu-manager-policy-options=""
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395689    2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395692    2554 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395694    2554 flags.go:64] FLAG: --enable-debugging-handlers="true"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395697    2554 flags.go:64] FLAG: --enable-load-reader="false"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395700    2554 flags.go:64] FLAG: --enable-server="true"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395702    2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395708    2554 flags.go:64] FLAG: --event-burst="100"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395711    2554 flags.go:64] FLAG: --event-qps="50"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395714    2554 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395717    2554 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395719    2554 flags.go:64] FLAG: --eviction-hard=""
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395723    2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395726    2554 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395729    2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Aug 04 18:00:21.401616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395733    2554 flags.go:64] FLAG: --eviction-soft=""
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395736    2554 flags.go:64] FLAG: --eviction-soft-grace-period=""
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395739    2554 flags.go:64] FLAG: --exit-on-lock-contention="false"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395741    2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395744    2554 flags.go:64] FLAG: --experimental-mounter-path=""
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395747    2554 flags.go:64] FLAG: --fail-cgroupv1="false"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395750    2554 flags.go:64] FLAG: --fail-swap-on="true"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395753    2554 flags.go:64] FLAG: --feature-gates=""
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395756    2554 flags.go:64] FLAG: --file-check-frequency="20s"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395759    2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395762    2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395765    2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395768    2554 flags.go:64] FLAG: --healthz-port="10248"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395771    2554 flags.go:64] FLAG: --help="false"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395775    2554 flags.go:64] FLAG: --hostname-override="ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395778    2554 flags.go:64] FLAG: --housekeeping-interval="10s"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395781    2554 flags.go:64] FLAG: --http-check-frequency="20s"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395783    2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395787    2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395790    2554 flags.go:64] FLAG: --image-gc-high-threshold="85"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395793    2554 flags.go:64] FLAG: --image-gc-low-threshold="80"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395795    2554 flags.go:64] FLAG: --image-service-endpoint=""
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395798    2554 flags.go:64] FLAG: --kernel-memcg-notification="false"
Aug 04 18:00:21.402295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395801    2554 flags.go:64] FLAG: --kube-api-burst="100"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395805    2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395808    2554 flags.go:64] FLAG: --kube-api-qps="50"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395810    2554 flags.go:64] FLAG: --kube-reserved=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395813    2554 flags.go:64] FLAG: --kube-reserved-cgroup=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395816    2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395819    2554 flags.go:64] FLAG: --kubelet-cgroups=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395822    2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395824    2554 flags.go:64] FLAG: --lock-file=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395827    2554 flags.go:64] FLAG: --log-cadvisor-usage="false"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395830    2554 flags.go:64] FLAG: --log-flush-frequency="5s"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395833    2554 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395838    2554 flags.go:64] FLAG: --log-json-split-stream="false"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395841    2554 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395843    2554 flags.go:64] FLAG: --log-text-split-stream="false"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395846    2554 flags.go:64] FLAG: --logging-format="text"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395849    2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395852    2554 flags.go:64] FLAG: --make-iptables-util-chains="true"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395855    2554 flags.go:64] FLAG: --manifest-url=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395857    2554 flags.go:64] FLAG: --manifest-url-header=""
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395861    2554 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395864    2554 flags.go:64] FLAG: --max-open-files="1000000"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395868    2554 flags.go:64] FLAG: --max-pods="110"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395872    2554 flags.go:64] FLAG: --maximum-dead-containers="-1"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395875    2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Aug 04 18:00:21.402878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395878    2554 flags.go:64] FLAG: --memory-manager-policy="None"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395881    2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395884    2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395887    2554 flags.go:64] FLAG: --node-ip="0.0.0.0"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395889    2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395896    2554 flags.go:64] FLAG: --node-status-max-images="50"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395899    2554 flags.go:64] FLAG: --node-status-update-frequency="10s"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395902    2554 flags.go:64] FLAG: --oom-score-adj="-999"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395905    2554 flags.go:64] FLAG: --pod-cidr=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395908    2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395913    2554 flags.go:64] FLAG: --pod-manifest-path=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395915    2554 flags.go:64] FLAG: --pod-max-pids="-1"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395918    2554 flags.go:64] FLAG: --pods-per-core="0"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395921    2554 flags.go:64] FLAG: --port="10250"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395924    2554 flags.go:64] FLAG: --protect-kernel-defaults="false"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395927    2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04d1c6117d6dfca19"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395930    2554 flags.go:64] FLAG: --qos-reserved=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395933    2554 flags.go:64] FLAG: --read-only-port="10255"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395936    2554 flags.go:64] FLAG: --register-node="true"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395939    2554 flags.go:64] FLAG: --register-with-taints=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395943    2554 flags.go:64] FLAG: --registry-burst="10"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395945    2554 flags.go:64] FLAG: --registry-qps="5"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395948    2554 flags.go:64] FLAG: --reserved-cpus=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395950    2554 flags.go:64] FLAG: --reserved-memory=""
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395954    2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Aug 04 18:00:21.403559 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395956    2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395959    2554 flags.go:64] FLAG: --rotate-certificates="false"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395962    2554 flags.go:64] FLAG: --rotate-server-certificates="false"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395964    2554 flags.go:64] FLAG: --runonce="false"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395967    2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395971    2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395974    2554 flags.go:64] FLAG: --seccomp-default="false"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395976    2554 flags.go:64] FLAG: --serialize-image-pulls="true"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395979    2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395982    2554 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395985    2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395988    2554 flags.go:64] FLAG: --storage-driver-password="root"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395990    2554 flags.go:64] FLAG: --storage-driver-secure="false"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395994    2554 flags.go:64] FLAG: --storage-driver-table="stats"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395997    2554 flags.go:64] FLAG: --storage-driver-user="root"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.395999    2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396002    2554 flags.go:64] FLAG: --sync-frequency="1m0s"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396005    2554 flags.go:64] FLAG: --system-cgroups=""
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396008    2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396013    2554 flags.go:64] FLAG: --system-reserved-cgroup=""
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396016    2554 flags.go:64] FLAG: --tls-cert-file=""
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396019    2554 flags.go:64] FLAG: --tls-cipher-suites="[]"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396023    2554 flags.go:64] FLAG: --tls-min-version=""
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396026    2554 flags.go:64] FLAG: --tls-private-key-file=""
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396028    2554 flags.go:64] FLAG: --topology-manager-policy="none"
Aug 04 18:00:21.404151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396031    2554 flags.go:64] FLAG: --topology-manager-policy-options=""
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396034    2554 flags.go:64] FLAG: --topology-manager-scope="container"
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396037    2554 flags.go:64] FLAG: --v="2"
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396041    2554 flags.go:64] FLAG: --version="false"
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396044    2554 flags.go:64] FLAG: --vmodule=""
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396048    2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396051    2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396172    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396177    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396181    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396185    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396188    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396192    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396195    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396197    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396200    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396203    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396205    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396208    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396210    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396227    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.404826 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396230    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396233    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396236    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396238    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396241    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396244    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396246    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396249    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396251    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396254    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396256    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396259    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396262    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396264    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396267    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396270    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396272    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396275    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396277    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.405381 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396281    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396284    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396286    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396289    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396293    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396295    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396298    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396300    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396303    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396305    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396308    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396310    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396316    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396318    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396320    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396323    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396326    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396328    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396331    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396333    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.405853 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396336    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396339    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396341    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396344    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396347    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396349    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396352    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396355    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396357    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396359    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396362    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396364    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396367    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396369    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396371    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396374    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396378    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396380    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396383    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396385    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.406390 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396388    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396392    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396394    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396397    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396400    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396403    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396406    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396408    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396411    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396414    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396416    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396419    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396421    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396423    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396426    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396430    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396433    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396436    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396439    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.406897 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396441    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396445    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396447    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396450    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396452    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396455    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.396457    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.396933    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.403423    2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.403440    2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.403461    2554 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.403467    2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403527    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403532    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.407482 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403535    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403538    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403542    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403544    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403547    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403550    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403553    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403555    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403558    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403560    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403563    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403566    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403570    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403572    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403576    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403578    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403581    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403583    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403586    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403589    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.407865 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403592    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403594    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403597    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403599    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403602    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403605    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403608    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403611    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403614    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403617    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403619    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403622    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403624    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403628    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403632    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403635    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403638    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403640    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403643    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.408428 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403645    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403648    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403650    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403653    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403655    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403658    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403660    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403663    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403666    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403668    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403671    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403673    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403676    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403679    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403681    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403685    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403689    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403691    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403694    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403697    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.408900 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403700    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403703    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403705    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403709    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403713    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403716    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403719    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403721    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403724    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403727    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403729    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403731    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403734    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403737    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403739    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403741    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403744    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403746    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403749    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403751    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.409441 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403754    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403756    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403758    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403762    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403765    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403767    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403770    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403773    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403775    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403778    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403780    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403783    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403786    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403789    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403791    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403794    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403796    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.409929 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403799    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.403804    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403915    2554 feature_gate.go:328] unrecognized feature gate: Example
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403919    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403922    2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403925    2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403928    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403930    2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403933    2554 feature_gate.go:328] unrecognized feature gate: OVNObservability
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403936    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403938    2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403941    2554 feature_gate.go:328] unrecognized feature gate: Example2
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403943    2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403946    2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403949    2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Aug 04 18:00:21.410448 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403951    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403954    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403957    2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403959    2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403962    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403966    2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403970    2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403973    2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403976    2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403978    2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403981    2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403983    2554 feature_gate.go:328] unrecognized feature gate: NewOLM
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403986    2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403988    2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403991    2554 feature_gate.go:328] unrecognized feature gate: SignatureStores
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403993    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403996    2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.403998    2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404001    2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404003    2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Aug 04 18:00:21.410837 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404006    2554 feature_gate.go:328] unrecognized feature gate: DualReplica
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404008    2554 feature_gate.go:328] unrecognized feature gate: OSStreams
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404011    2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404013    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404016    2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404019    2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404021    2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404023    2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404026    2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404028    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404031    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404033    2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404036    2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404039    2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404041    2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404043    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404046    2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404049    2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404051    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404053    2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Aug 04 18:00:21.411418 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404057    2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404059    2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404061    2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404064    2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404066    2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404069    2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404072    2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404074    2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404076    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404079    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404082    2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404084    2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404087    2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404091    2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404094    2554 feature_gate.go:328] unrecognized feature gate: PinnedImages
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404097    2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404100    2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404103    2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404105    2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Aug 04 18:00:21.412154 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404108    2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404111    2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404114    2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404116    2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404119    2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404122    2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404125    2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404127    2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404130    2554 feature_gate.go:328] unrecognized feature gate: EventTTL
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404133    2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404136    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404139    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404141    2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404144    2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404147    2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404149    2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404152    2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404155    2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404158    2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404160    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Aug 04 18:00:21.412674 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404163    2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404166    2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404168    2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404171    2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404173    2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404177    2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:21.404180    2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.404186    2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.404309    2554 server.go:962] "Client rotation is on, will bootstrap in background"
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.406241    2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.407429    2554 server.go:1019] "Starting client certificate rotation"
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.407532    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.407571    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Aug 04 18:00:21.413175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.410311    2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:21.417171 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.417142    2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Aug 04 18:00:21.427000 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.426977    2554 log.go:25] "Validated CRI v1 runtime API"
Aug 04 18:00:21.428995 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.428977    2554 log.go:25] "Validated CRI v1 image API"
Aug 04 18:00:21.431152 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.431136    2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Aug 04 18:00:21.434357 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.434326    2554 fs.go:135] Filesystem UUIDs: map[213d9784-1912-4f5b-9676-999d3bbe24cf:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a730ebf1-b242-47f0-b2bf-54420ff69e31:/dev/nvme0n1p3]
Aug 04 18:00:21.434426 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.434354    2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Aug 04 18:00:21.439332 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.439201    2554 manager.go:216] Machine: {Timestamp:2026-08-04 18:00:21.438056433 +0000 UTC m=+0.295557073 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107335 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21660d8b613113695902669fb0b982 SystemUUID:ec21660d-8b61-3113-6959-02669fb0b982 BootID:cafd48be-d325-4d2c-83bf-9760ea921800 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:66:5e:59:0f:7f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:66:5e:59:0f:7f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:3d:54:3f:ab:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Aug 04 18:00:21.439332 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.439326    2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Aug 04 18:00:21.439456 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.439410    2554 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Aug 04 18:00:21.440243 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.440210    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:21.442119 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.442088    2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Aug 04 18:00:21.442293 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.442121    2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-15.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Aug 04 18:00:21.443526 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.443516    2554 topology_manager.go:138] "Creating topology manager with none policy"
Aug 04 18:00:21.443570 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.443529    2554 container_manager_linux.go:309] "Creating device plugin manager"
Aug 04 18:00:21.443570 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.443538    2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:21.444051 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.444041    2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Aug 04 18:00:21.444781 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.444771    2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Aug 04 18:00:21.448586 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448563    2554 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:21.448698 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448688    2554 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Aug 04 18:00:21.448737 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448718    2554 kubelet.go:486] "Attempting to sync node with API server"
Aug 04 18:00:21.448737 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448728    2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Aug 04 18:00:21.448870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448742    2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Aug 04 18:00:21.448870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448752    2554 kubelet.go:398] "Adding apiserver pod source"
Aug 04 18:00:21.448870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.448762    2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Aug 04 18:00:21.449122 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.449110    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:21.449174 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.449129    2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Aug 04 18:00:21.451122 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.451103    2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Aug 04 18:00:21.452490 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.452477    2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Aug 04 18:00:21.452535 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.452508    2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Aug 04 18:00:21.453499 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453488    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453504    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453511    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453516    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453525    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453530    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Aug 04 18:00:21.453537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453538    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Aug 04 18:00:21.453694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453545    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Aug 04 18:00:21.453694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453552    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Aug 04 18:00:21.453694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453557    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Aug 04 18:00:21.453694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453572    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Aug 04 18:00:21.453694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.453577    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Aug 04 18:00:21.454153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.454144    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Aug 04 18:00:21.454153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.454153    2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Aug 04 18:00:21.458341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.458324    2554 server.go:1268] "Started kubelet"
Aug 04 18:00:21.458432 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.458407    2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Aug 04 18:00:21.458491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.458402    2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Aug 04 18:00:21.458491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.458465    2554 server_v1.go:49] "podresources" method="list" useActivePods=true
Aug 04 18:00:21.459400 ip-10-0-140-15 systemd[1]: Started Kubernetes Kubelet.
Aug 04 18:00:21.461365 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.461342    2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Aug 04 18:00:21.461658 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.461640    2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-15.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Aug 04 18:00:21.462095 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.462077    2554 server.go:310] "Adding debug handlers to kubelet server"
Aug 04 18:00:21.462319 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.462288    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Aug 04 18:00:21.462418 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.462288    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Aug 04 18:00:21.463544 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.463522    2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Aug 04 18:00:21.466111 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.465172    2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-15.ec2.internal.18c8ac0c6351c2bc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-15.ec2.internal,UID:ip-10-0-140-15.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-15.ec2.internal,},FirstTimestamp:2026-08-04 18:00:21.45828934 +0000 UTC m=+0.315789978,LastTimestamp:2026-08-04 18:00:21.45828934 +0000 UTC m=+0.315789978,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-15.ec2.internal,}"
Aug 04 18:00:21.466680 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.466663    2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Aug 04 18:00:21.466771 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.466683    2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:21.466771 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.466730    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.467253 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.467234    2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Aug 04 18:00:21.467336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.467237    2554 volume_manager.go:311] "The desired_state_of_world populator starts"
Aug 04 18:00:21.467336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.467270    2554 volume_manager.go:313] "Starting Kubelet Volume Manager"
Aug 04 18:00:21.467400 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.467369    2554 reconstruct.go:97] "Volume reconstruction finished"
Aug 04 18:00:21.467400 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.467377    2554 reconciler.go:29] "Reconciler: start to sync state"
Aug 04 18:00:21.468185 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468160    2554 factory.go:55] Registering systemd factory
Aug 04 18:00:21.468185 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468183    2554 factory.go:223] Registration of the systemd container factory successfully
Aug 04 18:00:21.468416 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468404    2554 factory.go:153] Registering CRI-O factory
Aug 04 18:00:21.468457 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468421    2554 factory.go:223] Registration of the crio container factory successfully
Aug 04 18:00:21.468491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468469    2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Aug 04 18:00:21.468521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468493    2554 factory.go:103] Registering Raw factory
Aug 04 18:00:21.468521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468506    2554 manager.go:1196] Started watching for new ooms in manager
Aug 04 18:00:21.468930 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.468909    2554 manager.go:349] Starting recovery of all containers
Aug 04 18:00:21.470314 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.470289    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7jmv6"
Aug 04 18:00:21.475989 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.475955    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Aug 04 18:00:21.477858 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.477785    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7jmv6"
Aug 04 18:00:21.477858 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.477800    2554 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-15.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Aug 04 18:00:21.478065 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.477925    2554 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Aug 04 18:00:21.479998 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.479981    2554 manager.go:354] Recovery completed
Aug 04 18:00:21.486367 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.486351    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.488383 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488363    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.488465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488399    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.488465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488411    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.488971 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488954    2554 cpu_manager.go:222] "Starting CPU manager" policy="none"
Aug 04 18:00:21.489034 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488972    2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Aug 04 18:00:21.489034 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488977    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.489034 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.488990    2554 state_mem.go:36] "Initialized new in-memory state store"
Aug 04 18:00:21.489967 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.489953    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.490018 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.489979    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.490018 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.489991    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.490324 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.490311    2554 policy_none.go:49] "None policy: Start"
Aug 04 18:00:21.490389 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.490331    2554 memory_manager.go:187] "Starting memorymanager" policy="None"
Aug 04 18:00:21.490389 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.490345    2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Aug 04 18:00:21.491470 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.491458    2554 policy_none.go:47] "Start"
Aug 04 18:00:21.532900 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.532693    2554 manager.go:337] "Starting Device Plugin manager"
Aug 04 18:00:21.533032 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.532933    2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Aug 04 18:00:21.533032 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.532947    2554 server.go:87] "Starting device plugin registration server"
Aug 04 18:00:21.533254 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.533240    2554 eviction_manager.go:189] "Eviction manager: starting control loop"
Aug 04 18:00:21.533342 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.533257    2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Aug 04 18:00:21.533513 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.533470    2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Aug 04 18:00:21.533630 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.533594    2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Aug 04 18:00:21.533630 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.533604    2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Aug 04 18:00:21.535200 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.534792    2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Aug 04 18:00:21.535200 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.534845    2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.633633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.633603    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.634739 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.634595    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.634739 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.634625    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.634739 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.634634    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.634739 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.634683    2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.643089 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.643042    2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.643089 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.643069    2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-15.ec2.internal\": node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.656949 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.656930    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.659814 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.659798    2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Aug 04 18:00:21.659898 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.659833    2554 status_manager.go:244] "Starting to sync pod status with apiserver"
Aug 04 18:00:21.659898 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.659850    2554 kubelet.go:2448] "Starting kubelet main sync loop"
Aug 04 18:00:21.659898 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.659879    2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Aug 04 18:00:21.661846 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.661831    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:21.757403 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.757368    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.760545 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.760512    2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"]
Aug 04 18:00:21.760618 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.760609    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.761628 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.761606    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.761749 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.761639    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.761749 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.761653    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.762980 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.762967    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.763082 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763066    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.763122 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763101    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.763615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763600    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.763715 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763631    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.763715 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763643    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.763715 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763699    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.763869 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763724    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.763869 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.763733    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.764696 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.764682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.764754 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.764705    2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Aug 04 18:00:21.765384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.765366    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientMemory"
Aug 04 18:00:21.765445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.765390    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasNoDiskPressure"
Aug 04 18:00:21.765445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.765404    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeHasSufficientPID"
Aug 04 18:00:21.768549 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.768528    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf92005c4ad1b4b9a80ced15063e26f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-15.ec2.internal\" (UID: \"bf92005c4ad1b4b9a80ced15063e26f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.768612 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.768569    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.768651 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.768616    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.781761 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.781741    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-15.ec2.internal\" not found" node="ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.785152 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.785137    2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-15.ec2.internal\" not found" node="ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.858465 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.858438    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:21.868848 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf92005c4ad1b4b9a80ced15063e26f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-15.ec2.internal\" (UID: \"bf92005c4ad1b4b9a80ced15063e26f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.868927 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868852    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.868927 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868870    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.868927 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868913    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.869054 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868974    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/db327713684dc4006fed9b22d9e32c2e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal\" (UID: \"db327713684dc4006fed9b22d9e32c2e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.869054 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:21.868976    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bf92005c4ad1b4b9a80ced15063e26f5-config\") pod \"kube-apiserver-proxy-ip-10-0-140-15.ec2.internal\" (UID: \"bf92005c4ad1b4b9a80ced15063e26f5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:21.958799 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:21.958716    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:22.059212 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:22.059172    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:22.084637 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:22.084610    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:22.087368 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:22.087351    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:22.159811 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:22.159775    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:23.731214 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:23.731147    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:23.831658 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:23.831625    2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-15.ec2.internal\" not found"
Aug 04 18:00:23.878181 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.878147    2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Aug 04 18:00:23.878578 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.878311    2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Aug 04 18:00:23.905530 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.905501    2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:23.920341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.920314    2554 apiserver.go:52] "Watching apiserver"
Aug 04 18:00:23.928818 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.928799    2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Aug 04 18:00:23.929572 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.929535    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6","openshift-cluster-node-tuning-operator/tuned-467z7","openshift-image-registry/node-ca-z9znh","openshift-network-diagnostics/network-check-target-wc9ks","openshift-network-operator/iptables-alerter-fqvgx","kube-system/global-pull-secret-syncer-hv2mv","openshift-dns/node-resolver-gfq2x","openshift-multus/multus-additional-cni-plugins-5xq9l","openshift-multus/multus-gwx7r","openshift-multus/network-metrics-daemon-hsmdl","openshift-ovn-kubernetes/ovnkube-node-njg2j","kube-system/konnectivity-agent-5n7p9"]
Aug 04 18:00:23.931367 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.931347    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.932337 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.932317    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.933330 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.933309    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:23.934007 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.933981    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.934007 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.933971    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Aug 04 18:00:23.934155 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934099    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4sgws\""
Aug 04 18:00:23.934155 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934130    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.934300 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934286    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:23.934418 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:23.934378    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:23.934860 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934845    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-ddwcv\""
Aug 04 18:00:23.934930 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934894    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.934930 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.934924    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.935320 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.935305    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:23.935488 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.935472    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.935577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.935519    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Aug 04 18:00:23.935577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.935549    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8s822\""
Aug 04 18:00:23.935673 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.935615    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.936404 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.936388    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:23.936518 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:23.936498    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:23.937217 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937203    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.937301 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937265    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.937358 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937306    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-mpjq8\""
Aug 04 18:00:23.937460 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937446    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:23.937563 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937549    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Aug 04 18:00:23.937633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937597    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:23.937633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.937619    2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Aug 04 18:00:23.938538 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.938523    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.939359 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.939340    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.939541 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.939525    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.939741 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.939716    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-md8rz\""
Aug 04 18:00:23.939818 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.939741    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:23.940670 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940653    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Aug 04 18:00:23.940760 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940678    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.940833 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940776    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.940833 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940778    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:23.940934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940862    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Aug 04 18:00:23.940934 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:23.940872    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:23.940934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.940880    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nl6qp\""
Aug 04 18:00:23.941099 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.941088    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Aug 04 18:00:23.941761 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.941737    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Aug 04 18:00:23.941906 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.941891    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8pqkl\""
Aug 04 18:00:23.942135 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942117    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:23.942280 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942104    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942366 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942288    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97hkf\" (UniqueName: \"kubernetes.io/projected/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kube-api-access-97hkf\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942366 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-sys\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942366 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942352    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-lib-modules\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942409    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjwqh\" (UniqueName: \"kubernetes.io/projected/f24ad3da-51c7-47b5-83fd-74528a20c094-kube-api-access-fjwqh\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942435    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca064d63-4126-457e-b8ba-240ef6524fc6-tmp-dir\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942460    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-system-cni-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942484    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942508    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.942553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942537    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5btjq\" (UniqueName: \"kubernetes.io/projected/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-kube-api-access-5btjq\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942559    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942582    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-device-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942617    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942649    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-tuned\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-tmp\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942694    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca064d63-4126-457e-b8ba-240ef6524fc6-hosts-file\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942731    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942751    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-conf\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-systemd\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942794    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-kubelet-config\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:23.942830 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942813    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-os-release\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942839    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942856    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-modprobe-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942869    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-run\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942883    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-host\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942896    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-cnibin\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9746\" (UniqueName: \"kubernetes.io/projected/0995f211-3886-4776-9e68-8c40f2a1cc0e-kube-api-access-g9746\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942952    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-host\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942965    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-serviceca\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.942979    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943017    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943017    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysconfig\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943079    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943108    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-dbus\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943132    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j94bw\" (UniqueName: \"kubernetes.io/projected/3902a019-e7cf-4612-b8f5-edb9db0d881c-kube-api-access-j94bw\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943174    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tg7\" (UniqueName: \"kubernetes.io/projected/ca064d63-4126-457e-b8ba-240ef6524fc6-kube-api-access-66tg7\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943198    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-kubernetes\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943219    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-var-lib-kubelet\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:23.943915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943242    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3902a019-e7cf-4612-b8f5-edb9db0d881c-host-slash\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:23.943915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943267    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:23.943915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943290    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:23.943915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.943314    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3902a019-e7cf-4612-b8f5-edb9db0d881c-iptables-alerter-script\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:23.944085 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944017    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Aug 04 18:00:23.944153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944098    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Aug 04 18:00:23.944225 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944212    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Aug 04 18:00:23.944517 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944488    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Aug 04 18:00:23.944599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944552    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Aug 04 18:00:23.944599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944584    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kg7f4\""
Aug 04 18:00:23.944706 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.944603    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Aug 04 18:00:23.945820 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.945081    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Aug 04 18:00:23.945820 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.945248    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-5kvpk\""
Aug 04 18:00:23.945820 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.945401    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Aug 04 18:00:23.947787 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.947769    2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Aug 04 18:00:23.949294 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.949278    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:23.949385 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.949336    2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"
Aug 04 18:00:23.949448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.949384    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal"]
Aug 04 18:00:23.952838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.952366    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-08-03 17:55:22 +0000 UTC" deadline="2028-02-28 12:29:35.555074573 +0000 UTC"
Aug 04 18:00:23.952838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.952407    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13746h29m11.602672383s"
Aug 04 18:00:23.955938 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.955916    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal"]
Aug 04 18:00:23.955938 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.955927    2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Aug 04 18:00:23.977118 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.977091    2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jcwrx"
Aug 04 18:00:23.985895 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:23.985827    2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jcwrx"
Aug 04 18:00:24.038942 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.038909    2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Aug 04 18:00:24.043682 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043661    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-multus\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.043770 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043692    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.043770 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-tmp\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.043770 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.043875 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043771    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-agent-certs\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.043875 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043801    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-systemd-units\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.043875 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043841    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-socket-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.043982 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043907    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-netns\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.043982 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043929    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca064d63-4126-457e-b8ba-240ef6524fc6-hosts-file\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.043982 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043948    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-kubelet-config\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.043982 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043973    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.044237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043985    2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Aug 04 18:00:24.044237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-host\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.044237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044076    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-kubelet\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.044237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044107    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2rd2\" (UniqueName: \"kubernetes.io/projected/03456812-0405-4710-965d-a544cc7be2ef-kube-api-access-f2rd2\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.044237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.043984    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ca064d63-4126-457e-b8ba-240ef6524fc6-hosts-file\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.044442 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044192    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-kubelet-config\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.044442 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044232    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-host\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.044442 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044202    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-script-lib\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.044576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044558    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.044621 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044558    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.044731 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044718    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-modprobe-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.044763 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044745    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-run\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.044799 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044775    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-cnibin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.044831 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044805    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-konnectivity-ca\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.044831 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044821    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysconfig\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.044934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044854    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.044934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044877    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0100536f-3d05-49fd-973c-1d04c047f8d6-ovn-node-metrics-cert\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.044934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044899    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tg7\" (UniqueName: \"kubernetes.io/projected/ca064d63-4126-457e-b8ba-240ef6524fc6-kube-api-access-66tg7\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.044934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044903    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysconfig\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.044934 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044923    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3902a019-e7cf-4612-b8f5-edb9db0d881c-host-slash\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044945    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044957    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-os-release\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044965    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3902a019-e7cf-4612-b8f5-edb9db0d881c-host-slash\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044964    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-run\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044985    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-k8s-cni-cncf-io\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.044909    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-modprobe-d\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045056    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-node-log\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045092    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045124    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6hsm\" (UniqueName: \"kubernetes.io/projected/0100536f-3d05-49fd-973c-1d04c047f8d6-kube-api-access-j6hsm\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045141    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3902a019-e7cf-4612-b8f5-edb9db0d881c-iptables-alerter-script\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045156    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-netns\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045170    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhnh\" (UniqueName: \"kubernetes.io/projected/4bea9f6f-a036-4ccb-bc75-5214012b9a70-kube-api-access-9zhnh\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045187    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-slash\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045220    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045289    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-env-overrides\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045316    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97hkf\" (UniqueName: \"kubernetes.io/projected/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kube-api-access-97hkf\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045332    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-sys\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045348    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-lib-modules\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045370    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjwqh\" (UniqueName: \"kubernetes.io/projected/f24ad3da-51c7-47b5-83fd-74528a20c094-kube-api-access-fjwqh\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045386    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045394    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-sys\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045404    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-kubelet\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045500    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-lib-modules\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045577    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-system-cni-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045619    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045641    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-system-cni-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045645    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045686    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-socket-dir-parent\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045711    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-ovn\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045735    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-bin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045776    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045782    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-device-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/3902a019-e7cf-4612-b8f5-edb9db0d881c-iptables-alerter-script\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045824    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045855    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-sys-fs\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.045941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045857    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-tuned\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045886    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-system-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045899    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-device-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045914    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045935    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-systemd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045966    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.045994    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046016    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-conf\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046072    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-systemd\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-registration-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046093    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-os-release\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046140    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-multus-certs\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046153    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-os-release\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046158    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-sysctl-conf\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046168    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-host\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046174    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-systemd\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046189    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0995f211-3886-4776-9e68-8c40f2a1cc0e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.046560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046202    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-cnibin\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.047086 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.046207    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-host\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.048963 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.048939    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9746\" (UniqueName: \"kubernetes.io/projected/0995f211-3886-4776-9e68-8c40f2a1cc0e-kube-api-access-g9746\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.049126 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049093    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0995f211-3886-4776-9e68-8c40f2a1cc0e-cnibin\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.049238 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049218    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-tuned\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.049311 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049221    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f24ad3da-51c7-47b5-83fd-74528a20c094-tmp\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.049497 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049478    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-serviceca\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.049579 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049525    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-var-lib-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.049579 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049560    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-log-socket\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.049707 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049597    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.049707 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049647    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-dbus\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.049707 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049699    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j94bw\" (UniqueName: \"kubernetes.io/projected/3902a019-e7cf-4612-b8f5-edb9db0d881c-kube-api-access-j94bw\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.049843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049732    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-hostroot\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.049843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049733    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.049843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049752    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-serviceca\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.049843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049754    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/290108f0-7082-4602-827e-23aa0cc5015c-dbus\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.050019 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049861    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-multus-daemon-config\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.050019 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049937    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-kubernetes\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.050019 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.049970    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-var-lib-kubelet\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.050019 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:24.050216 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050016    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-etc-kubernetes\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.050216 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050060    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.050216 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050075    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f24ad3da-51c7-47b5-83fd-74528a20c094-var-lib-kubelet\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.050216 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050094    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-etc-kubernetes\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.050216 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-netd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.050425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050293    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-config\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.050425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.050425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050365    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-conf-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.050425 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.050294    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:24.050425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050392    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-bin\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.050629 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050424    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca064d63-4126-457e-b8ba-240ef6524fc6-tmp-dir\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.050629 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050475    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ee8cd94-3e65-47b1-b964-d7cc188c833e-etc-selinux\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.050629 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050467    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5btjq\" (UniqueName: \"kubernetes.io/projected/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-kube-api-access-5btjq\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.050753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050628    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-cni-binary-copy\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.050753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050677    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-etc-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.050842 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.050778    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca064d63-4126-457e-b8ba-240ef6524fc6-tmp-dir\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.050842 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.050795    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.550758032 +0000 UTC m=+1.937448764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:24.054708 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.054688    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjwqh\" (UniqueName: \"kubernetes.io/projected/f24ad3da-51c7-47b5-83fd-74528a20c094-kube-api-access-fjwqh\") pod \"tuned-467z7\" (UID: \"f24ad3da-51c7-47b5-83fd-74528a20c094\") " pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.054835 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.054813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97hkf\" (UniqueName: \"kubernetes.io/projected/2ee8cd94-3e65-47b1-b964-d7cc188c833e-kube-api-access-97hkf\") pod \"aws-ebs-csi-driver-node-q9wd6\" (UID: \"2ee8cd94-3e65-47b1-b964-d7cc188c833e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.055119 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.055102    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tg7\" (UniqueName: \"kubernetes.io/projected/ca064d63-4126-457e-b8ba-240ef6524fc6-kube-api-access-66tg7\") pod \"node-resolver-gfq2x\" (UID: \"ca064d63-4126-457e-b8ba-240ef6524fc6\") " pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.057167 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.057150    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:24.057261 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.057172    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:24.057261 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.057185    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:24.057261 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.057181    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9746\" (UniqueName: \"kubernetes.io/projected/0995f211-3886-4776-9e68-8c40f2a1cc0e-kube-api-access-g9746\") pod \"multus-additional-cni-plugins-5xq9l\" (UID: \"0995f211-3886-4776-9e68-8c40f2a1cc0e\") " pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.057261 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.057251    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.557235912 +0000 UTC m=+1.943926643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:24.059074 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.059019    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j94bw\" (UniqueName: \"kubernetes.io/projected/3902a019-e7cf-4612-b8f5-edb9db0d881c-kube-api-access-j94bw\") pod \"iptables-alerter-fqvgx\" (UID: \"3902a019-e7cf-4612-b8f5-edb9db0d881c\") " pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.059590 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.059564    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5btjq\" (UniqueName: \"kubernetes.io/projected/3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea-kube-api-access-5btjq\") pod \"node-ca-z9znh\" (UID: \"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea\") " pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.151381 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-etc-kubernetes\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151381 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151381    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-netd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151396    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-config\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151411    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-conf-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-bin\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-cni-binary-copy\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-etc-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151461    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-netd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151486    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-etc-kubernetes\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151497    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-conf-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151525    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-etc-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151541    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-cni-bin\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151564    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-multus\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151599    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-agent-certs\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151615    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-systemd-units\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.151642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151632    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-netns\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151655    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-multus\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151671    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-kubelet\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151678    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-systemd-units\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151713    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-netns\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151720    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2rd2\" (UniqueName: \"kubernetes.io/projected/03456812-0405-4710-965d-a544cc7be2ef-kube-api-access-f2rd2\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151761    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-script-lib\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151795    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-cnibin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151819    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-konnectivity-ca\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151845    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0100536f-3d05-49fd-973c-1d04c047f8d6-ovn-node-metrics-cert\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151870    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-os-release\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151943    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-kubelet\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151989    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-k8s-cni-cncf-io\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152014    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-node-log\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.151924    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-cnibin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152058    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152091    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-cni-binary-copy\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152095    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-config\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.152336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152120    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-run-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152132    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-os-release\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6hsm\" (UniqueName: \"kubernetes.io/projected/0100536f-3d05-49fd-973c-1d04c047f8d6-kube-api-access-j6hsm\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152151    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-k8s-cni-cncf-io\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152175    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-netns\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152178    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-node-log\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152220    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-netns\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152286    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhnh\" (UniqueName: \"kubernetes.io/projected/4bea9f6f-a036-4ccb-bc75-5214012b9a70-kube-api-access-9zhnh\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-slash\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152374    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152398    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-env-overrides\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152412    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-slash\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152460    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152554    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152629    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-kubelet\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152695    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-socket-dir-parent\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152719    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-ovn\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152742    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-bin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153222 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.152755    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-system-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152781    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-konnectivity-ca\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152802    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152802    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-ovnkube-script-lib\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.152824    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:24.652805723 +0000 UTC m=+2.039496438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152859    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152872    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-multus-socket-dir-parent\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152862    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-var-lib-cni-bin\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-kubelet\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152891    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-ovn\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152822    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0100536f-3d05-49fd-973c-1d04c047f8d6-env-overrides\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.152941    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-systemd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-system-cni-dir\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153059    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-run-systemd\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153090    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.153730 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153146    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-multus-certs\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153196    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-host-run-multus-certs\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153217    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-var-lib-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-log-socket\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-hostroot\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153312    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-multus-daemon-config\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153524    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-log-socket\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153549    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0100536f-3d05-49fd-973c-1d04c047f8d6-var-lib-openvswitch\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.153572    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/03456812-0405-4710-965d-a544cc7be2ef-hostroot\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.154165    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/03456812-0405-4710-965d-a544cc7be2ef-multus-daemon-config\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.154502 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.154404    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0100536f-3d05-49fd-973c-1d04c047f8d6-ovn-node-metrics-cert\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.154502 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.154407    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/46df8a4b-d4c2-4a7a-ac67-705e1445e84b-agent-certs\") pod \"konnectivity-agent-5n7p9\" (UID: \"46df8a4b-d4c2-4a7a-ac67-705e1445e84b\") " pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.162595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.162563    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2rd2\" (UniqueName: \"kubernetes.io/projected/03456812-0405-4710-965d-a544cc7be2ef-kube-api-access-f2rd2\") pod \"multus-gwx7r\" (UID: \"03456812-0405-4710-965d-a544cc7be2ef\") " pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.164409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.164384    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6hsm\" (UniqueName: \"kubernetes.io/projected/0100536f-3d05-49fd-973c-1d04c047f8d6-kube-api-access-j6hsm\") pod \"ovnkube-node-njg2j\" (UID: \"0100536f-3d05-49fd-973c-1d04c047f8d6\") " pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.164657 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.164641    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhnh\" (UniqueName: \"kubernetes.io/projected/4bea9f6f-a036-4ccb-bc75-5214012b9a70-kube-api-access-9zhnh\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.174842 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.174818    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb327713684dc4006fed9b22d9e32c2e.slice/crio-62ace66d867a9958380dd04cceb96fc717c4ee114076bda2d891b30a504922b8 WatchSource:0}: Error finding container 62ace66d867a9958380dd04cceb96fc717c4ee114076bda2d891b30a504922b8: Status 404 returned error can't find the container with id 62ace66d867a9958380dd04cceb96fc717c4ee114076bda2d891b30a504922b8
Aug 04 18:00:24.180185 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.180167    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:00:24.256916 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.256832    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6"
Aug 04 18:00:24.265561 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.265538    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee8cd94_3e65_47b1_b964_d7cc188c833e.slice/crio-2ebef100910ae9db98db9172263ae4a98ac5b98af6dc724a200c5ba1c2718da1 WatchSource:0}: Error finding container 2ebef100910ae9db98db9172263ae4a98ac5b98af6dc724a200c5ba1c2718da1: Status 404 returned error can't find the container with id 2ebef100910ae9db98db9172263ae4a98ac5b98af6dc724a200c5ba1c2718da1
Aug 04 18:00:24.286465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.286431    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-467z7"
Aug 04 18:00:24.293000 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.292973    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-z9znh"
Aug 04 18:00:24.294459 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.294438    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf24ad3da_51c7_47b5_83fd_74528a20c094.slice/crio-40a218c5b1bbbd224e1460fffac54f306b99aa581dcd117d7b6ff713203d855e WatchSource:0}: Error finding container 40a218c5b1bbbd224e1460fffac54f306b99aa581dcd117d7b6ff713203d855e: Status 404 returned error can't find the container with id 40a218c5b1bbbd224e1460fffac54f306b99aa581dcd117d7b6ff713203d855e
Aug 04 18:00:24.302022 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.301998    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cee3f9c_fdf7_4d1c_ac51_bf894c2b36ea.slice/crio-e6dc49ce3e6cf283a75fb34776fb998be472f30367282687bdaa2b7fc3ba82ea WatchSource:0}: Error finding container e6dc49ce3e6cf283a75fb34776fb998be472f30367282687bdaa2b7fc3ba82ea: Status 404 returned error can't find the container with id e6dc49ce3e6cf283a75fb34776fb998be472f30367282687bdaa2b7fc3ba82ea
Aug 04 18:00:24.306953 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.306935    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fqvgx"
Aug 04 18:00:24.314465 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.314441    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3902a019_e7cf_4612_b8f5_edb9db0d881c.slice/crio-9094a89298762afd1908d1387da68da094d672e9277c46b16cf4ebb68c083875 WatchSource:0}: Error finding container 9094a89298762afd1908d1387da68da094d672e9277c46b16cf4ebb68c083875: Status 404 returned error can't find the container with id 9094a89298762afd1908d1387da68da094d672e9277c46b16cf4ebb68c083875
Aug 04 18:00:24.325500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.325475    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gfq2x"
Aug 04 18:00:24.333973 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.333133    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca064d63_4126_457e_b8ba_240ef6524fc6.slice/crio-c7e088e87e6278166626b59950a65ccf81120f5263e13a3b9c809ca2d821f70c WatchSource:0}: Error finding container c7e088e87e6278166626b59950a65ccf81120f5263e13a3b9c809ca2d821f70c: Status 404 returned error can't find the container with id c7e088e87e6278166626b59950a65ccf81120f5263e13a3b9c809ca2d821f70c
Aug 04 18:00:24.340838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.340819    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xq9l"
Aug 04 18:00:24.348017 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.347997    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0995f211_3886_4776_9e68_8c40f2a1cc0e.slice/crio-c6b099989a62292891d98dfe089ba3461889b89ab6530929f5d28016c79f9d57 WatchSource:0}: Error finding container c6b099989a62292891d98dfe089ba3461889b89ab6530929f5d28016c79f9d57: Status 404 returned error can't find the container with id c6b099989a62292891d98dfe089ba3461889b89ab6530929f5d28016c79f9d57
Aug 04 18:00:24.366558 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.366534    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gwx7r"
Aug 04 18:00:24.374285 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.374261    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03456812_0405_4710_965d_a544cc7be2ef.slice/crio-baa695edb747410b0362677d82be698553690536b52507099a421168a876024c WatchSource:0}: Error finding container baa695edb747410b0362677d82be698553690536b52507099a421168a876024c: Status 404 returned error can't find the container with id baa695edb747410b0362677d82be698553690536b52507099a421168a876024c
Aug 04 18:00:24.376190 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.376170    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:24.383314 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.380633    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:24.388078 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.388049    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0100536f_3d05_49fd_973c_1d04c047f8d6.slice/crio-5a7c58383ffba6d0943fda105cc638e3640f4a16bc8f6fea95d7bb738735d986 WatchSource:0}: Error finding container 5a7c58383ffba6d0943fda105cc638e3640f4a16bc8f6fea95d7bb738735d986: Status 404 returned error can't find the container with id 5a7c58383ffba6d0943fda105cc638e3640f4a16bc8f6fea95d7bb738735d986
Aug 04 18:00:24.391736 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.391700    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46df8a4b_d4c2_4a7a_ac67_705e1445e84b.slice/crio-d44459dc017461a68416d08ab16c352ea54b6fb172264731b02e719b7974f315 WatchSource:0}: Error finding container d44459dc017461a68416d08ab16c352ea54b6fb172264731b02e719b7974f315: Status 404 returned error can't find the container with id d44459dc017461a68416d08ab16c352ea54b6fb172264731b02e719b7974f315
Aug 04 18:00:24.417910 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:24.417880    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf92005c4ad1b4b9a80ced15063e26f5.slice/crio-1b24fe4c801f06f5d58fc0460a20e888a680c0864546b7dc9dc84b0346590e47 WatchSource:0}: Error finding container 1b24fe4c801f06f5d58fc0460a20e888a680c0864546b7dc9dc84b0346590e47: Status 404 returned error can't find the container with id 1b24fe4c801f06f5d58fc0460a20e888a680c0864546b7dc9dc84b0346590e47
Aug 04 18:00:24.424555 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.424537    2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:24.449767 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.449746    2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:24.556118 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.556022    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:24.556258 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.556141    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:24.556258 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.556200    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.55618426 +0000 UTC m=+2.942874971 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:24.657335 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.657302    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:24.657504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.657393    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:24.657504 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657468    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:24.657504 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657487    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:24.657504 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657496    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:24.657673 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657518    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:24.657673 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657545    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.657529919 +0000 UTC m=+3.044220633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:24.657673 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:24.657568    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:25.65755203 +0000 UTC m=+3.044242744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:24.863909 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.863828    2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Aug 04 18:00:24.987903 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.987837    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:23 +0000 UTC" deadline="2028-04-06 08:40:03.966240358 +0000 UTC"
Aug 04 18:00:24.987903 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:24.987899    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14654h39m38.978345558s"
Aug 04 18:00:25.140566 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.140367    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal" event={"ID":"bf92005c4ad1b4b9a80ced15063e26f5","Type":"ContainerStarted","Data":"1b24fe4c801f06f5d58fc0460a20e888a680c0864546b7dc9dc84b0346590e47"}
Aug 04 18:00:25.145717 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.145622    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5n7p9" event={"ID":"46df8a4b-d4c2-4a7a-ac67-705e1445e84b","Type":"ContainerStarted","Data":"d44459dc017461a68416d08ab16c352ea54b6fb172264731b02e719b7974f315"}
Aug 04 18:00:25.148124 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.148003    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"5a7c58383ffba6d0943fda105cc638e3640f4a16bc8f6fea95d7bb738735d986"}
Aug 04 18:00:25.149753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.149679    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gwx7r" event={"ID":"03456812-0405-4710-965d-a544cc7be2ef","Type":"ContainerStarted","Data":"baa695edb747410b0362677d82be698553690536b52507099a421168a876024c"}
Aug 04 18:00:25.154464 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.154223    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfq2x" event={"ID":"ca064d63-4126-457e-b8ba-240ef6524fc6","Type":"ContainerStarted","Data":"c7e088e87e6278166626b59950a65ccf81120f5263e13a3b9c809ca2d821f70c"}
Aug 04 18:00:25.160126 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.160071    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9znh" event={"ID":"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea","Type":"ContainerStarted","Data":"e6dc49ce3e6cf283a75fb34776fb998be472f30367282687bdaa2b7fc3ba82ea"}
Aug 04 18:00:25.161861 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.161836    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerStarted","Data":"c6b099989a62292891d98dfe089ba3461889b89ab6530929f5d28016c79f9d57"}
Aug 04 18:00:25.164090 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.164066    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqvgx" event={"ID":"3902a019-e7cf-4612-b8f5-edb9db0d881c","Type":"ContainerStarted","Data":"9094a89298762afd1908d1387da68da094d672e9277c46b16cf4ebb68c083875"}
Aug 04 18:00:25.165496 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.165434    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-467z7" event={"ID":"f24ad3da-51c7-47b5-83fd-74528a20c094","Type":"ContainerStarted","Data":"40a218c5b1bbbd224e1460fffac54f306b99aa581dcd117d7b6ff713203d855e"}
Aug 04 18:00:25.166772 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.166686    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6" event={"ID":"2ee8cd94-3e65-47b1-b964-d7cc188c833e","Type":"ContainerStarted","Data":"2ebef100910ae9db98db9172263ae4a98ac5b98af6dc724a200c5ba1c2718da1"}
Aug 04 18:00:25.168218 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.168162    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal" event={"ID":"db327713684dc4006fed9b22d9e32c2e","Type":"ContainerStarted","Data":"62ace66d867a9958380dd04cceb96fc717c4ee114076bda2d891b30a504922b8"}
Aug 04 18:00:25.563763 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.563665    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:25.563910 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.563868    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:25.563974 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.563959    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:27.563941373 +0000 UTC m=+4.950632095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:25.664664 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.664625    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:25.664899 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.664712    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:25.664899 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.664890    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:25.665077 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.664910    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:25.665077 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.664933    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:25.665077 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.664992    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:27.66497392 +0000 UTC m=+5.051664633 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:25.665478 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.665433    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:25.665575 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:25.665493    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:27.665475732 +0000 UTC m=+5.052166448 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:25.988868 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.988788    2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-08-03 17:55:23 +0000 UTC" deadline="2028-01-20 23:22:00.880696529 +0000 UTC"
Aug 04 18:00:25.988868 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:25.988829    2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12821h21m34.891871329s"
Aug 04 18:00:26.131096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:26.130998    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:26.131252 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:26.131154    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:26.131633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:26.131596    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:26.131739 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:26.131690    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:26.131803 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:26.131772    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:26.131875 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:26.131858    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:27.580138 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:27.579475    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:27.580138 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.579668    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:27.580138 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.579745    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.579715316 +0000 UTC m=+8.966406028 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:27.680753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:27.680716    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:27.680941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:27.680785    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:27.681002 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.680958    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:27.681002 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.680980    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:27.681002 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.680992    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:27.681224 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.681077    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.681050223 +0000 UTC m=+9.067740935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:27.681458 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.681441    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:27.681523 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:27.681486    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:31.681474609 +0000 UTC m=+9.068165320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:28.131314 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:28.131276    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:28.131509 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:28.131487    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:28.131568 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:28.131550    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:28.131602 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:28.131595    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:28.131642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:28.131627    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:28.131679 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:28.131668    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:30.130910 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:30.130875    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:30.131339 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:30.131018    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:30.131428 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:30.131403    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:30.131558 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:30.131508    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:30.131634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:30.131560    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:30.131677 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:30.131629    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:31.615045 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:31.614990    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:31.615530 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.615142    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.615530 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.615208    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:39.615187018 +0000 UTC m=+17.001877738 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:31.715598 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:31.715559    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:31.715795 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:31.715624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:31.715795 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.715780    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:31.715921 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.715812    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:31.715921 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.715826    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.715921 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.715883    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:39.715865065 +0000 UTC m=+17.102555782 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:31.716312 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.716293    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:31.716394 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:31.716337    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:39.716325075 +0000 UTC m=+17.103015785 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:32.131201    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:32.131350    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:32.131682    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:32.131790    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:32.131833    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:32.131950 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:32.131917    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:34.131808 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:34.131767    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:34.132276 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:34.131818    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:34.132276 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:34.131783    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:34.132276 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:34.131911    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:34.132276 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:34.131993    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:34.132276 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:34.132106    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:36.131869 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:36.131833    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:36.131869 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:36.131863    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:36.132377 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:36.131950    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:36.132377 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:36.131980    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:36.132377 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:36.132073    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:36.132377 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:36.132178    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:38.131867 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:38.131834    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:38.132337 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:38.131882    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:38.132337 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:38.131953    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:38.132337 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:38.131970    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:38.132337 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:38.132079    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:38.132337 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:38.132161    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:39.675472 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:39.675432    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:39.675963 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.675600    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:39.675963 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.675682    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:00:55.675660166 +0000 UTC m=+33.062350894 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:39.775993 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:39.775955    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:39.776197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:39.776023    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:39.776197 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776135    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:39.776282 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776201    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:55.77618511 +0000 UTC m=+33.162875821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:39.776282 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776214    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:39.776282 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776230    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:39.776282 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776242    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:39.776412 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:39.776290    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:55.776276016 +0000 UTC m=+33.162966741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:40.131684 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:40.131608    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:40.131684 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:40.131641    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:40.131936 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:40.131608    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:40.131936 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:40.131762    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:40.131936 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:40.131895    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:40.132115 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:40.131991    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:42.131051 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:42.130998    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:42.131051 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:42.131018    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:42.131650 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:42.131001    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:42.131650 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:42.131140    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:42.131650 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:42.131202    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:42.131650 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:42.131297    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:44.131131 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:44.131103    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:44.131599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:44.131104    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:44.131599 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:44.131235    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:44.131599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:44.131104    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:44.131599 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:44.131303    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:44.131599 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:44.131358    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:45.219909 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.219505    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-467z7" event={"ID":"f24ad3da-51c7-47b5-83fd-74528a20c094","Type":"ContainerStarted","Data":"a8c661622eaeee601dd83dd35583139aff5de0b761cc8610a9417075e5b8ba4d"}
Aug 04 18:00:45.221260 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.221229    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal" event={"ID":"bf92005c4ad1b4b9a80ced15063e26f5","Type":"ContainerStarted","Data":"8c8f54e37552158c9bbc02531f0aac9984b05df9d6aff7d825c3055ca2e0cc8b"}
Aug 04 18:00:45.222686 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.222607    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"7a3fe77db44e13aee4505c998e80f058d6b273652ed59002fab154f53f5b2697"}
Aug 04 18:00:45.225987 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.225965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gwx7r" event={"ID":"03456812-0405-4710-965d-a544cc7be2ef","Type":"ContainerStarted","Data":"934740957c33483d45cb8e970cc06d1ded0a3b93a9b44c515c60a74c1ca59e88"}
Aug 04 18:00:45.246853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.246802    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-467z7" podStartSLOduration=1.749797997 podStartE2EDuration="22.246788963s" totalImagesPullingTime="20.496990966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:45.246331495 +0000 UTC m=+22.633022228" watchObservedRunningTime="2026-08-04 18:00:45.246788963 +0000 UTC m=+22.633479695"
Aug 04 18:00:45.269489 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.269207    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gwx7r" podStartSLOduration=1.850221597 podStartE2EDuration="22.26918838s" totalImagesPullingTime="20.418966783s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:45.268826158 +0000 UTC m=+22.655516891" watchObservedRunningTime="2026-08-04 18:00:45.26918838 +0000 UTC m=+22.655879116"
Aug 04 18:00:45.290299 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:45.290244    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-15.ec2.internal" podStartSLOduration=22.29022561 podStartE2EDuration="22.29022561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:45.289823319 +0000 UTC m=+22.676514035" watchObservedRunningTime="2026-08-04 18:00:45.29022561 +0000 UTC m=+22.676916346"
Aug 04 18:00:46.131117 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.131079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:46.131117 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.131117    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:46.131345 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:46.131179    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:46.131345 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.131079    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:46.131345 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:46.131271    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:46.131345 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:46.131339    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:46.229235 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.229198    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="a990e56b137bd895766d7631f76348a2cc3bffd97c666457fb547579ca2721f5" exitCode=0
Aug 04 18:00:46.229951 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.229293    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"a990e56b137bd895766d7631f76348a2cc3bffd97c666457fb547579ca2721f5"}
Aug 04 18:00:46.230906 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.230886    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6" event={"ID":"2ee8cd94-3e65-47b1-b964-d7cc188c833e","Type":"ContainerStarted","Data":"31b4d666985dddb1d3492d64fd24ca61a215d9d80ac1a38584affd6421eb628b"}
Aug 04 18:00:46.232138 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.232116    2554 generic.go:356] "Generic (PLEG): container finished" podID="db327713684dc4006fed9b22d9e32c2e" containerID="93b090c37292485438cbee82bef932be5b2320ea1113d4d48b5bfb8b73204aae" exitCode=0
Aug 04 18:00:46.232232 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.232169    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal" event={"ID":"db327713684dc4006fed9b22d9e32c2e","Type":"ContainerDied","Data":"93b090c37292485438cbee82bef932be5b2320ea1113d4d48b5bfb8b73204aae"}
Aug 04 18:00:46.241105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.240960    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5n7p9" event={"ID":"46df8a4b-d4c2-4a7a-ac67-705e1445e84b","Type":"ContainerStarted","Data":"17eabad336a26ddfde11e4f64b655ee6eaace295a288249af20018c4a70dff77"}
Aug 04 18:00:46.243751 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.243730    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"b8cf93d27f508e26f6f9abfdd94c37f0ddd0116996fa6d11a1adda9c21fff817"}
Aug 04 18:00:46.243834 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.243787    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"003f42a1a4e4528b821954233b2a71602507b3807681ca255c5b43f0b120bcd7"}
Aug 04 18:00:46.243834 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.243826    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"c8bf8b8e4bd4c445fe0bd236a52b083a512a0b25afb5c6dc61cba0e46dd1a791"}
Aug 04 18:00:46.243938 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.243870    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"27891c3c4c93360c52640e2df744479ba2d996cafdfb7e7ff8f7f422487504dc"}
Aug 04 18:00:46.243938 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.243901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"665ab997039e00ae364554979141fa00b03bfc4c436cfb674a22696f15ac0d2b"}
Aug 04 18:00:46.244958 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.244934    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gfq2x" event={"ID":"ca064d63-4126-457e-b8ba-240ef6524fc6","Type":"ContainerStarted","Data":"6b4e85ca1b7b956a85b271f63ec2bc2a9c6014ccda3cfa0760a7da1cd7174e45"}
Aug 04 18:00:46.246130 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.246106    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-z9znh" event={"ID":"3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea","Type":"ContainerStarted","Data":"8b37ad2068c0ba8148cb125f968525dbf065304dd0370700fe50f9cc0ee294f4"}
Aug 04 18:00:46.282843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.282803    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-z9znh" podStartSLOduration=2.85705323 podStartE2EDuration="23.282788138s" totalImagesPullingTime="20.425734908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.282364137 +0000 UTC m=+23.669054870" watchObservedRunningTime="2026-08-04 18:00:46.282788138 +0000 UTC m=+23.669478871"
Aug 04 18:00:46.298272 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.298236    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gfq2x" podStartSLOduration=2.932517477 podStartE2EDuration="23.29822127s" totalImagesPullingTime="20.365703793s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.29798648 +0000 UTC m=+23.684677212" watchObservedRunningTime="2026-08-04 18:00:46.29822127 +0000 UTC m=+23.684912002"
Aug 04 18:00:46.313940 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.313902    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5n7p9" podStartSLOduration=3.006040279 podStartE2EDuration="23.313886803s" totalImagesPullingTime="20.307846524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:46.313501849 +0000 UTC m=+23.700192582" watchObservedRunningTime="2026-08-04 18:00:46.313886803 +0000 UTC m=+23.700577538"
Aug 04 18:00:46.387146 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.386966    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:46.457889 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:46.457858    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.250284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.250243    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fqvgx" event={"ID":"3902a019-e7cf-4612-b8f5-edb9db0d881c","Type":"ContainerStarted","Data":"a1606f4b05a9d86e564e822c2dc90e37f5683dbd8db7e398be93a604bd38f8ec"}
Aug 04 18:00:47.252717 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.252307    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal" event={"ID":"db327713684dc4006fed9b22d9e32c2e","Type":"ContainerStarted","Data":"3650685b023e3657a5989d3ff83ee2a5629efdcea0c2be7ef92772e2b080b3e9"}
Aug 04 18:00:47.265654 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.265605    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fqvgx" podStartSLOduration=3.782075614 podStartE2EDuration="24.265587623s" totalImagesPullingTime="20.483512009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:47.265082782 +0000 UTC m=+24.651773514" watchObservedRunningTime="2026-08-04 18:00:47.265587623 +0000 UTC m=+24.652278359"
Aug 04 18:00:47.281518 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.281461    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-15.ec2.internal" podStartSLOduration=24.28144068 podStartE2EDuration="24.28144068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:47.280772019 +0000 UTC m=+24.667462752" watchObservedRunningTime="2026-08-04 18:00:47.28144068 +0000 UTC m=+24.668131416"
Aug 04 18:00:47.494769 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.494737    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:47.575547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:47.575428    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:48.131415 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:48.131356    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:48.131598 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:48.131432    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:48.131598 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:48.131466    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:48.131598 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:48.131551    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:48.131911 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:48.131883    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:48.132002 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:48.131979    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:48.258195 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:48.258164    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"38bab77ee24884fd558c130e978f99be072ac997adac50643679c83d42009d1d"}
Aug 04 18:00:48.394865 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:48.394841    2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Aug 04 18:00:49.019980 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:49.019848    2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-08-04T18:00:48.394858622Z","UUID":"5a09a824-7ce1-4158-9966-647c68dd8395","Handler":null,"Name":"","Endpoint":""}
Aug 04 18:00:49.021309 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:49.021273    2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Aug 04 18:00:49.021432 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:49.021318    2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Aug 04 18:00:49.265449 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:49.265410    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6" event={"ID":"2ee8cd94-3e65-47b1-b964-d7cc188c833e","Type":"ContainerStarted","Data":"736c20a9c57ab44ca2ad704062f7cbd52e0a9bf4832ab721b5539decd1d96241"}
Aug 04 18:00:50.131135 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:50.131101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:50.131310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:50.131143    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:50.131310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:50.131101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:50.131310 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:50.131239    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:50.131448 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:50.131335    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:50.131448 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:50.131421    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:50.859639 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:50.859412    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:50.860099 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:50.859946    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:51.271395 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.271369    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="e4be4d464caefb1720011c9f25cdb6744261c05b4a5a73e4c92d1185f1821598" exitCode=0
Aug 04 18:00:51.271502 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.271454    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"e4be4d464caefb1720011c9f25cdb6744261c05b4a5a73e4c92d1185f1821598"}
Aug 04 18:00:51.275705 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.275678    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" event={"ID":"0100536f-3d05-49fd-973c-1d04c047f8d6","Type":"ContainerStarted","Data":"6ed5bc19d5cb6f962548befce5348da54368040aabc56e0614664131efaf26a2"}
Aug 04 18:00:51.276427 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.276326    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:51.276427 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.276397    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:51.276665 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.276648    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5n7p9"
Aug 04 18:00:51.293244 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.293218    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:51.317871 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.317821    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j" podStartSLOduration=7.659853636 podStartE2EDuration="28.317806814s" totalImagesPullingTime="20.657953178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:51.317339788 +0000 UTC m=+28.704030520" watchObservedRunningTime="2026-08-04 18:00:51.317806814 +0000 UTC m=+28.704497546"
Aug 04 18:00:51.470238 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:51.470151    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:52.131181 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.131151    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:52.131181 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.131168    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:52.131872 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.131172    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:52.131872 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.131253    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:52.131872 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.131318    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:52.131872 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.131382    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:52.281386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.281355    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6" event={"ID":"2ee8cd94-3e65-47b1-b964-d7cc188c833e","Type":"ContainerStarted","Data":"ee7bfa3ebfe53b5c3701c3fbab7c3740d10d0bdaafc0255cb5dbc330e9fd71ad"}
Aug 04 18:00:52.283387 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.283363    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:52.283492 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.283442    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:52.303449 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.303380    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-q9wd6" podStartSLOduration=3.515846355 podStartE2EDuration="29.303361047s" totalImagesPullingTime="25.787514692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:00:52.303015231 +0000 UTC m=+29.689705965" watchObservedRunningTime="2026-08-04 18:00:52.303361047 +0000 UTC m=+29.690051782"
Aug 04 18:00:52.312998 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.312972    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:00:52.674298 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.674117    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hv2mv"]
Aug 04 18:00:52.674463 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.674372    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:52.674504 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.674459    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:52.679900 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.679812    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsmdl"]
Aug 04 18:00:52.680177 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.680109    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:52.680245 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.680222    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:52.680536 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.680517    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wc9ks"]
Aug 04 18:00:52.680647 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:52.680585    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:52.680705 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:52.680657    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:53.285591 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:53.285559    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="e2961fef0288f4184e99cbac21ca106a6f4fca130f54435e68b3b869abc101f5" exitCode=0
Aug 04 18:00:53.286062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:53.285662    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"e2961fef0288f4184e99cbac21ca106a6f4fca130f54435e68b3b869abc101f5"}
Aug 04 18:00:54.131163 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:54.131130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:54.131163 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:54.131160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:54.131374 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:54.131160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:54.131374 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:54.131261    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:54.131374 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:54.131332    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:54.131524 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:54.131377    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:55.292538 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:55.292503    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="9cc77067eecd31852eb61ff83ceb555c1bd82593f5f74312df867e4d505548c1" exitCode=0
Aug 04 18:00:55.292538 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:55.292542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"9cc77067eecd31852eb61ff83ceb555c1bd82593f5f74312df867e4d505548c1"}
Aug 04 18:00:55.699796 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:55.699763    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:55.699965 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.699861    2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:55.699965 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.699923    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret podName:290108f0-7082-4602-827e-23aa0cc5015c nodeName:}" failed. No retries permitted until 2026-08-04 18:01:27.699898881 +0000 UTC m=+65.086589592 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret") pod "global-pull-secret-syncer-hv2mv" (UID: "290108f0-7082-4602-827e-23aa0cc5015c") : object "kube-system"/"original-pull-secret" not registered
Aug 04 18:00:55.800490 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:55.800453    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:55.800512    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800612    2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800614    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800637    2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800646    2554 projected.go:196] Error preparing data for projected volume kube-api-access-dqfkd for pod openshift-network-diagnostics/network-check-target-wc9ks: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:55.800662 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800660    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs podName:4bea9f6f-a036-4ccb-bc75-5214012b9a70 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:27.800647889 +0000 UTC m=+65.187338599 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs") pod "network-metrics-daemon-hsmdl" (UID: "4bea9f6f-a036-4ccb-bc75-5214012b9a70") : object "openshift-multus"/"metrics-daemon-secret" not registered
Aug 04 18:00:55.800875 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:55.800689    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd podName:69a6ac8e-d331-4b39-9046-19ad6bead317 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:27.800675203 +0000 UTC m=+65.187365914 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dqfkd" (UniqueName: "kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd") pod "network-check-target-wc9ks" (UID: "69a6ac8e-d331-4b39-9046-19ad6bead317") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Aug 04 18:00:56.131659 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:56.131628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:56.131810 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:56.131628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:56.131810 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:56.131743    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:56.131810 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:56.131628    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:56.131974 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:56.131811    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:56.131974 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:56.131900    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:57.962215 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:57.962180    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:58.033144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.033111    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:58.131114 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.131083    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:00:58.131258 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.131119    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:00:58.131258 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.131090    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:00:58.131258 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:58.131204    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-hsmdl" podUID="4bea9f6f-a036-4ccb-bc75-5214012b9a70"
Aug 04 18:00:58.131356 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:58.131263    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc9ks" podUID="69a6ac8e-d331-4b39-9046-19ad6bead317"
Aug 04 18:00:58.131356 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:58.131318    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hv2mv" podUID="290108f0-7082-4602-827e-23aa0cc5015c"
Aug 04 18:00:58.947264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.947236    2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-15.ec2.internal" event="NodeReady"
Aug 04 18:00:58.947410 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:58.947342    2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Aug 04 18:00:59.000783 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.000747    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cbzvp"]
Aug 04 18:00:59.002805 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.002788    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mz9vp"]
Aug 04 18:00:59.002933 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.002918    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.004312 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.004297    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.005370 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.005351    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Aug 04 18:00:59.005460 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.005415    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Aug 04 18:00:59.005517 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.005415    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2cqb\""
Aug 04 18:00:59.006406 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.006390    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Aug 04 18:00:59.006469 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.006413    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Aug 04 18:00:59.006521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.006499    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rzh2g\""
Aug 04 18:00:59.006762 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.006743    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Aug 04 18:00:59.014773 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.014752    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz9vp"]
Aug 04 18:00:59.015616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.015592    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbzvp"]
Aug 04 18:00:59.123491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.123491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123493    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5658ae88-67e4-4545-ae55-5a352c56be47-config-volume\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.123694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123610    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/74d02d17-8838-4cfc-b54c-4c0e85fd2093-kube-api-access-nxplt\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.123694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123648    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74d02d17-8838-4cfc-b54c-4c0e85fd2093-cert\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.123768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123692    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5658ae88-67e4-4545-ae55-5a352c56be47-tmp-dir\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.123768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.123720    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx44c\" (UniqueName: \"kubernetes.io/projected/5658ae88-67e4-4545-ae55-5a352c56be47-kube-api-access-bx44c\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.142196 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.142163    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:00:59.224659 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224572    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/74d02d17-8838-4cfc-b54c-4c0e85fd2093-kube-api-access-nxplt\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.224659 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224613    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74d02d17-8838-4cfc-b54c-4c0e85fd2093-cert\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.224837 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224792    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5658ae88-67e4-4545-ae55-5a352c56be47-tmp-dir\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.224891 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224833    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx44c\" (UniqueName: \"kubernetes.io/projected/5658ae88-67e4-4545-ae55-5a352c56be47-kube-api-access-bx44c\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.224891 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224877    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.224991 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.224914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5658ae88-67e4-4545-ae55-5a352c56be47-config-volume\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.225069 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:59.224990    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:59.225130 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:59.225074    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:00:59.725052866 +0000 UTC m=+37.111743581 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:00:59.225235 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.225199    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5658ae88-67e4-4545-ae55-5a352c56be47-tmp-dir\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.225472 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.225456    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5658ae88-67e4-4545-ae55-5a352c56be47-config-volume\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.229322 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.229301    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/74d02d17-8838-4cfc-b54c-4c0e85fd2093-cert\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.236684 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.236555    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx44c\" (UniqueName: \"kubernetes.io/projected/5658ae88-67e4-4545-ae55-5a352c56be47-kube-api-access-bx44c\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.236684 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.236678    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/74d02d17-8838-4cfc-b54c-4c0e85fd2093-kube-api-access-nxplt\") pod \"ingress-canary-mz9vp\" (UID: \"74d02d17-8838-4cfc-b54c-4c0e85fd2093\") " pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.322867 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.322835    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mz9vp"
Aug 04 18:00:59.483196 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.482813    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mz9vp"]
Aug 04 18:00:59.487881 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:00:59.487851    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d02d17_8838_4cfc_b54c_4c0e85fd2093.slice/crio-29fc319d514233d81d0791a824b902e58a6fd757c9d70b596a9177baa9336714 WatchSource:0}: Error finding container 29fc319d514233d81d0791a824b902e58a6fd757c9d70b596a9177baa9336714: Status 404 returned error can't find the container with id 29fc319d514233d81d0791a824b902e58a6fd757c9d70b596a9177baa9336714
Aug 04 18:00:59.728384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:00:59.728344    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:00:59.728582 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:59.728518    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:00:59.728639 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:00:59.728596    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:00.728575906 +0000 UTC m=+38.115266627 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:00.131277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.131240    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:01:00.131956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.131423    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:01:00.131956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.131802    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:01:00.135231 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135211    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:00.135373 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135354    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:00.135451 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135374    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:00.135451 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135386    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4qc\""
Aug 04 18:01:00.135451 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135408    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:00.135908 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.135743    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nfg9c\""
Aug 04 18:01:00.214713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.214680    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:00.306232 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.306194    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz9vp" event={"ID":"74d02d17-8838-4cfc-b54c-4c0e85fd2093","Type":"ContainerStarted","Data":"29fc319d514233d81d0791a824b902e58a6fd757c9d70b596a9177baa9336714"}
Aug 04 18:01:00.737810 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:00.737776    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:00.737980 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:00.737924    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:00.738065 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:00.737987    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:02.737968651 +0000 UTC m=+40.124659361 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.756662 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:02.756621    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:02.757365 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:02.756808    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:02.757365 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:02.756877    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:06.756858228 +0000 UTC m=+44.143548938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:04.103105 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.103066    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.204583 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.204554    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.317009 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.316967    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mz9vp" event={"ID":"74d02d17-8838-4cfc-b54c-4c0e85fd2093","Type":"ContainerStarted","Data":"38155bad59fbe2fbf3854e5933812a9eff5811b5436383c3345353c5466f6d71"}
Aug 04 18:01:04.319576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.319550    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="c8f15769c9b74863026835e28e084c061bdfb8145483910a969b00aa1e49a365" exitCode=0
Aug 04 18:01:04.319691 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.319601    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"c8f15769c9b74863026835e28e084c061bdfb8145483910a969b00aa1e49a365"}
Aug 04 18:01:04.334715 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.334673    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mz9vp" podStartSLOduration=1.878014285 podStartE2EDuration="6.334658964s" totalImagesPullingTime="4.456644679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:04.333928107 +0000 UTC m=+41.720618842" watchObservedRunningTime="2026-08-04 18:01:04.334658964 +0000 UTC m=+41.721349696"
Aug 04 18:01:04.481996 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.481971    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:04.582277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:04.582251    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:05.324987 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:05.324950    2554 generic.go:356] "Generic (PLEG): container finished" podID="0995f211-3886-4776-9e68-8c40f2a1cc0e" containerID="9d76fcf25441ba612586d17338781ae77620ea3d4ea8abca3c6f35b1c90f3be7" exitCode=0
Aug 04 18:01:05.325437 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:05.325058    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerDied","Data":"9d76fcf25441ba612586d17338781ae77620ea3d4ea8abca3c6f35b1c90f3be7"}
Aug 04 18:01:06.330174 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:06.330130    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" event={"ID":"0995f211-3886-4776-9e68-8c40f2a1cc0e","Type":"ContainerStarted","Data":"a1a23690670ef5327d15acebd5ab0950f8c34d65790b0644c7e7acf182fd2be0"}
Aug 04 18:01:06.354517 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:06.354473    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5xq9l" podStartSLOduration=8.207136856 podStartE2EDuration="43.354458178s" totalImagesPullingTime="35.147321322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:06.353102605 +0000 UTC m=+43.739793337" watchObservedRunningTime="2026-08-04 18:01:06.354458178 +0000 UTC m=+43.741148912"
Aug 04 18:01:06.486288 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:06.486256    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:06.550933 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:06.550892    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:06.786911 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:06.786835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:06.787072 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:06.786980    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:06.787072 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:06.787065    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:14.787047617 +0000 UTC m=+52.173738327 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:09.890642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.890602    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"]
Aug 04 18:01:09.921813 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.921781    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"
Aug 04 18:01:09.936924 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.936894    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"]
Aug 04 18:01:09.937595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.937572    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Aug 04 18:01:09.937911 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.937890    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xghzs\""
Aug 04 18:01:09.940495 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:09.940468    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.002824 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.002797    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"]
Aug 04 18:01:10.025396 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.025372    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"]
Aug 04 18:01:10.025511 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.025458    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.028197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.028139    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Aug 04 18:01:10.028317 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.028214    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Aug 04 18:01:10.028488 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.028381    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.028904 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.028458    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Aug 04 18:01:10.028904 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.028635    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-hkhrr\""
Aug 04 18:01:10.104520 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.104486    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-64dcfc95cb-lqxzk"]
Aug 04 18:01:10.111831 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.111804    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnd22\" (UniqueName: \"kubernetes.io/projected/35dc7b5d-4102-49e7-9fbf-3dd75627a2eb-kube-api-access-jnd22\") pod \"volume-data-source-validator-5b8776669-9sbvq\" (UID: \"35dc7b5d-4102-49e7-9fbf-3dd75627a2eb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"
Aug 04 18:01:10.123008 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.122989    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64dcfc95cb-lqxzk"]
Aug 04 18:01:10.123152 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.123121    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.128002 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.127977    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Aug 04 18:01:10.128118 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.128054    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Aug 04 18:01:10.128470 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.128449    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Aug 04 18:01:10.128643 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.128629    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vw6kw\""
Aug 04 18:01:10.128791 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.128771    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Aug 04 18:01:10.128960 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.128943    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Aug 04 18:01:10.132225 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.132210    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Aug 04 18:01:10.212735 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.212662    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/754f774e-53e8-4b98-bea2-8c64eeea8f28-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.212868 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.212738    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnd22\" (UniqueName: \"kubernetes.io/projected/35dc7b5d-4102-49e7-9fbf-3dd75627a2eb-kube-api-access-jnd22\") pod \"volume-data-source-validator-5b8776669-9sbvq\" (UID: \"35dc7b5d-4102-49e7-9fbf-3dd75627a2eb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"
Aug 04 18:01:10.212868 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.212770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/754f774e-53e8-4b98-bea2-8c64eeea8f28-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.212868 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.212792    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqfs\" (UniqueName: \"kubernetes.io/projected/754f774e-53e8-4b98-bea2-8c64eeea8f28-kube-api-access-ppqfs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.222341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.222314    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnd22\" (UniqueName: \"kubernetes.io/projected/35dc7b5d-4102-49e7-9fbf-3dd75627a2eb-kube-api-access-jnd22\") pod \"volume-data-source-validator-5b8776669-9sbvq\" (UID: \"35dc7b5d-4102-49e7-9fbf-3dd75627a2eb\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"
Aug 04 18:01:10.230098 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.230077    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"
Aug 04 18:01:10.298351 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.298257    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.313566 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313532    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqfs\" (UniqueName: \"kubernetes.io/projected/754f774e-53e8-4b98-bea2-8c64eeea8f28-kube-api-access-ppqfs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.313679 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313612    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/754f774e-53e8-4b98-bea2-8c64eeea8f28-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.313679 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313650    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.313760 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313678    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4gjw\" (UniqueName: \"kubernetes.io/projected/9429ca73-a4a0-41a3-a154-5426e63032c9-kube-api-access-t4gjw\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.313760 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313710    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-default-certificate\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.313760 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313752    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.313909 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-stats-auth\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.313961 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.313914    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/754f774e-53e8-4b98-bea2-8c64eeea8f28-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.314321 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.314303    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/754f774e-53e8-4b98-bea2-8c64eeea8f28-config\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.316009 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.315988    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/754f774e-53e8-4b98-bea2-8c64eeea8f28-serving-cert\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.323264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.323243    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqfs\" (UniqueName: \"kubernetes.io/projected/754f774e-53e8-4b98-bea2-8c64eeea8f28-kube-api-access-ppqfs\") pod \"kube-storage-version-migrator-operator-7878f4b88c-x789v\" (UID: \"754f774e-53e8-4b98-bea2-8c64eeea8f28\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.337447 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.337420    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"
Aug 04 18:01:10.363814 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.363788    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:10.372694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.372668    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq"]
Aug 04 18:01:10.374739 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:10.374704    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35dc7b5d_4102_49e7_9fbf_3dd75627a2eb.slice/crio-8a4e4da194a75f752d20977a349188465f33cf580b84ab99ae1ab2ba725284ff WatchSource:0}: Error finding container 8a4e4da194a75f752d20977a349188465f33cf580b84ab99ae1ab2ba725284ff: Status 404 returned error can't find the container with id 8a4e4da194a75f752d20977a349188465f33cf580b84ab99ae1ab2ba725284ff
Aug 04 18:01:10.414319 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.414289    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.414319 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.414321    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4gjw\" (UniqueName: \"kubernetes.io/projected/9429ca73-a4a0-41a3-a154-5426e63032c9-kube-api-access-t4gjw\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.414492 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.414341    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-default-certificate\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.414492 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.414449    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:10.414586 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.414553    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.414636 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.414602    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-stats-auth\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.414702 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.414687    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.914663945 +0000 UTC m=+48.301354659 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : secret "router-metrics-certs-default" not found
Aug 04 18:01:10.414807 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.414793    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:10.914778613 +0000 UTC m=+48.301469860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:10.417956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.417081    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-default-certificate\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.419046 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.418873    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-stats-auth\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.423580 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.423562    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4gjw\" (UniqueName: \"kubernetes.io/projected/9429ca73-a4a0-41a3-a154-5426e63032c9-kube-api-access-t4gjw\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.497247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.497142    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v"]
Aug 04 18:01:10.500397 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:10.500372    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754f774e_53e8_4b98_bea2_8c64eeea8f28.slice/crio-b00aa1c38a71b330bdde77c112afb71d12d7a744e44817bfe34017da4104a373 WatchSource:0}: Error finding container b00aa1c38a71b330bdde77c112afb71d12d7a744e44817bfe34017da4104a373: Status 404 returned error can't find the container with id b00aa1c38a71b330bdde77c112afb71d12d7a744e44817bfe34017da4104a373
Aug 04 18:01:10.917883 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.917850    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.918355 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:10.917900    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:10.918355 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.918015    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:10.918355 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.918083    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:11.918064362 +0000 UTC m=+49.304755096 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:10.918355 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:10.918104    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:11.918094295 +0000 UTC m=+49.304785007 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : secret "router-metrics-certs-default" not found
Aug 04 18:01:11.343360 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:11.343318    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v" event={"ID":"754f774e-53e8-4b98-bea2-8c64eeea8f28","Type":"ContainerStarted","Data":"b00aa1c38a71b330bdde77c112afb71d12d7a744e44817bfe34017da4104a373"}
Aug 04 18:01:11.344627 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:11.344595    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq" event={"ID":"35dc7b5d-4102-49e7-9fbf-3dd75627a2eb","Type":"ContainerStarted","Data":"8a4e4da194a75f752d20977a349188465f33cf580b84ab99ae1ab2ba725284ff"}
Aug 04 18:01:11.926041 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:11.925994    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:11.926446 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:11.926055    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:11.926446 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:11.926166    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:11.926446 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:11.926176    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:13.926160329 +0000 UTC m=+51.312851040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:11.926446 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:11.926247    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:13.92623325 +0000 UTC m=+51.312923961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : secret "router-metrics-certs-default" not found
Aug 04 18:01:12.348785 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:12.348753    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq" event={"ID":"35dc7b5d-4102-49e7-9fbf-3dd75627a2eb","Type":"ContainerStarted","Data":"33e232e12f71a05c524017f9c69f957634b348794469077371ec078343f870f0"}
Aug 04 18:01:12.368258 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:12.368211    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5b8776669-9sbvq" podStartSLOduration=2.025656597 podStartE2EDuration="3.368193383s" totalImagesPullingTime="1.342536786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:12.367088208 +0000 UTC m=+49.753778940" watchObservedRunningTime="2026-08-04 18:01:12.368193383 +0000 UTC m=+49.754884116"
Aug 04 18:01:12.454647 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:12.454614    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:12.562380 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:12.562349    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:13.942304 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:13.942261    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:13.942733 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:13.942355    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:13.942733 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:13.942435    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:17.942417215 +0000 UTC m=+55.329107927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:13.942733 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:13.942440    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:13.942733 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:13.942476    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:17.942468615 +0000 UTC m=+55.329159327 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : secret "router-metrics-certs-default" not found
Aug 04 18:01:14.356410 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:14.356374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v" event={"ID":"754f774e-53e8-4b98-bea2-8c64eeea8f28","Type":"ContainerStarted","Data":"8c89244d2bd8074ff5f45643266d3619aa9fc58a4ef9ee8963797e3c14319a73"}
Aug 04 18:01:14.375356 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:14.375291    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v" podStartSLOduration=1.733289455 podStartE2EDuration="5.375272956s" totalImagesPullingTime="3.641983501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:14.373908456 +0000 UTC m=+51.760599189" watchObservedRunningTime="2026-08-04 18:01:14.375272956 +0000 UTC m=+51.761963690"
Aug 04 18:01:14.815876 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:14.815836    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:14.849460 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:14.849432    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:14.849587 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:14.849562    2554 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Aug 04 18:01:14.849631 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:14.849619    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls podName:5658ae88-67e4-4545-ae55-5a352c56be47 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.849604428 +0000 UTC m=+68.236295139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls") pod "dns-default-cbzvp" (UID: "5658ae88-67e4-4545-ae55-5a352c56be47") : secret "dns-default-metrics-tls" not found
Aug 04 18:01:14.906611 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:14.906574    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.091441 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.091363    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gwkr7"]
Aug 04 18:01:15.109445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.109410    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.115525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.115500    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwkr7"]
Aug 04 18:01:15.115634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.115607    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.118329 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.118305    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Aug 04 18:01:15.118449 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.118351    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Aug 04 18:01:15.118504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.118455    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Aug 04 18:01:15.118573 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.118559    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vx672\""
Aug 04 18:01:15.119179 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.119163    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Aug 04 18:01:15.152477 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.152453    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52fba404-4a23-4553-803a-0a1c47fc074d-data-volume\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.152595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.152509    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtbh4\" (UniqueName: \"kubernetes.io/projected/52fba404-4a23-4553-803a-0a1c47fc074d-kube-api-access-rtbh4\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.152595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.152561    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.152595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.152578    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52fba404-4a23-4553-803a-0a1c47fc074d-crio-socket\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.152690 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.152654    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.183478 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.183453    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:15.253843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.253820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.253947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.253853    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52fba404-4a23-4553-803a-0a1c47fc074d-crio-socket\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.253947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.253878    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.253947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.253898    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52fba404-4a23-4553-803a-0a1c47fc074d-data-volume\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.253947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.253943    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtbh4\" (UniqueName: \"kubernetes.io/projected/52fba404-4a23-4553-803a-0a1c47fc074d-kube-api-access-rtbh4\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.254173 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.253955    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.254173 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.254013    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.254173 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.254050    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:15.754011071 +0000 UTC m=+53.140701782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.254173 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.254089    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:15.754071802 +0000 UTC m=+53.140762527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.254173 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.254092    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/52fba404-4a23-4553-803a-0a1c47fc074d-crio-socket\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.254374 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.254355    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/52fba404-4a23-4553-803a-0a1c47fc074d-data-volume\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.264685 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.264658    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtbh4\" (UniqueName: \"kubernetes.io/projected/52fba404-4a23-4553-803a-0a1c47fc074d-kube-api-access-rtbh4\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.758411 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.758357    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.758411 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:15.758425    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:15.758670 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.758527    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.758670 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.758598    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:15.758670 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.758626    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.758604859 +0000 UTC m=+54.145295587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:15.758821 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:15.758682    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:16.758668716 +0000 UTC m=+54.145359426 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:16.742085 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.742052    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"]
Aug 04 18:01:16.766841 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.766805    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:16.766992 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.766851    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:16.766992 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:16.766922    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:16.766992 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:16.766938    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:16.767157 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:16.767005    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.766984243 +0000 UTC m=+56.153674953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:16.767157 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:16.767047    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:18.767020719 +0000 UTC m=+56.153711432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:16.770227 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.770211    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"]
Aug 04 18:01:16.770309 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.770299    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"
Aug 04 18:01:16.772980 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.772963    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Aug 04 18:01:16.773846 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.773831    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Aug 04 18:01:16.774906 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.774887    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-wkcvk\""
Aug 04 18:01:16.844122 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.844098    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfq2x_ca064d63-4126-457e-b8ba-240ef6524fc6/dns-node-resolver/0.log"
Aug 04 18:01:16.867525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.867494    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzllg\" (UniqueName: \"kubernetes.io/projected/7bf910e7-72eb-4fb2-a499-7c69ee708b20-kube-api-access-lzllg\") pod \"migrator-68bc745d87-t2dfj\" (UID: \"7bf910e7-72eb-4fb2-a499-7c69ee708b20\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"
Aug 04 18:01:16.968725 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.968688    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzllg\" (UniqueName: \"kubernetes.io/projected/7bf910e7-72eb-4fb2-a499-7c69ee708b20-kube-api-access-lzllg\") pod \"migrator-68bc745d87-t2dfj\" (UID: \"7bf910e7-72eb-4fb2-a499-7c69ee708b20\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"
Aug 04 18:01:16.978601 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:16.978575    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzllg\" (UniqueName: \"kubernetes.io/projected/7bf910e7-72eb-4fb2-a499-7c69ee708b20-kube-api-access-lzllg\") pod \"migrator-68bc745d87-t2dfj\" (UID: \"7bf910e7-72eb-4fb2-a499-7c69ee708b20\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"
Aug 04 18:01:17.079178 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.079148    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"
Aug 04 18:01:17.218678 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.218567    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj"]
Aug 04 18:01:17.221122 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:17.221093    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf910e7_72eb_4fb2_a499_7c69ee708b20.slice/crio-81e7aa176b4f55c331a4b1fb5ee2e0fe8a5ae182b050ea32f2ac30cc2f182734 WatchSource:0}: Error finding container 81e7aa176b4f55c331a4b1fb5ee2e0fe8a5ae182b050ea32f2ac30cc2f182734: Status 404 returned error can't find the container with id 81e7aa176b4f55c331a4b1fb5ee2e0fe8a5ae182b050ea32f2ac30cc2f182734
Aug 04 18:01:17.264080 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.264042    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.360964 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.360883    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:17.363755 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.363724    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj" event={"ID":"7bf910e7-72eb-4fb2-a499-7c69ee708b20","Type":"ContainerStarted","Data":"81e7aa176b4f55c331a4b1fb5ee2e0fe8a5ae182b050ea32f2ac30cc2f182734"}
Aug 04 18:01:17.973980 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.973884    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:17.974476 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:17.974051    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:17.974476 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:17.974080    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:25.974060972 +0000 UTC m=+63.360751696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:17.974476 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:17.974160    2554 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Aug 04 18:01:17.974476 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:17.974247    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:25.97422968 +0000 UTC m=+63.360920392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : secret "router-metrics-certs-default" not found
Aug 04 18:01:18.244063 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:18.243953    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9znh_3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea/node-ca/0.log"
Aug 04 18:01:18.366128 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:18.366094    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:18.780611 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:18.780565    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:18.780611 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:18.780624    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:18.780856 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:18.780695    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:18.780856 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:18.780714    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:18.780856 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:18.780767    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:22.780744613 +0000 UTC m=+60.167435347 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:18.780856 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:18.780785    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:22.78077659 +0000 UTC m=+60.167467301 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:19.045997 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:19.045918    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz9vp_74d02d17-8838-4cfc-b54c-4c0e85fd2093/serve-healthcheck-canary/0.log"
Aug 04 18:01:19.370553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:19.370522    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj" event={"ID":"7bf910e7-72eb-4fb2-a499-7c69ee708b20","Type":"ContainerStarted","Data":"c790f97a7c9fc5ccdc89ff4cf4d02f33ce195046f47a2b6ad54a7e62b869dd4f"}
Aug 04 18:01:19.370737 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:19.370569    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj" event={"ID":"7bf910e7-72eb-4fb2-a499-7c69ee708b20","Type":"ContainerStarted","Data":"cb8528f0845fee36186512bd81436d27fb7d4d26a7e8aece519181c2bb3a9cb5"}
Aug 04 18:01:19.392807 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:19.392754    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-t2dfj" podStartSLOduration=1.9991872590000002 podStartE2EDuration="3.392736986s" totalImagesPullingTime="1.393549727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:19.3881715 +0000 UTC m=+56.774862234" watchObservedRunningTime="2026-08-04 18:01:19.392736986 +0000 UTC m=+56.779427753"
Aug 04 18:01:20.285042 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:20.284985    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.386409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:20.386372    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.563041 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:20.563005    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.622967 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:20.622934    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:20.742968 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:20.742931    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.031434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:21.031334    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:21.784943 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:21.784909    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.177651 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:22.177615    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.374372 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:22.374332    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:22.815156 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:22.815131    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:22.815518 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:22.815173    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:22.815518 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:22.815261    2554 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Aug 04 18:01:22.815518 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:22.815273    2554 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:22.815518 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:22.815322    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.815309021 +0000 UTC m=+68.201999733 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : secret "insights-runtime-extractor-tls" not found
Aug 04 18:01:22.815518 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:22.815335    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm podName:52fba404-4a23-4553-803a-0a1c47fc074d nodeName:}" failed. No retries permitted until 2026-08-04 18:01:30.815329433 +0000 UTC m=+68.202020143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gwkr7" (UID: "52fba404-4a23-4553-803a-0a1c47fc074d") : configmap "kube-rbac-proxy" not found
Aug 04 18:01:22.903675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:22.903628    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.007525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:23.007491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.185460 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:23.185362    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.589560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:23.589525    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:23.986298 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:23.986215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.104338 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:24.104300    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:24.303990 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:24.303916    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-njg2j"
Aug 04 18:01:26.039455 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:26.039414    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:26.039854 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:26.039470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:26.039854 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:01:26.039587    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle podName:9429ca73-a4a0-41a3-a154-5426e63032c9 nodeName:}" failed. No retries permitted until 2026-08-04 18:01:42.039574062 +0000 UTC m=+79.426264772 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle") pod "router-default-64dcfc95cb-lqxzk" (UID: "9429ca73-a4a0-41a3-a154-5426e63032c9") : configmap references non-existent config key: service-ca.crt
Aug 04 18:01:26.041963 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:26.041944    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9429ca73-a4a0-41a3-a154-5426e63032c9-metrics-certs\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:26.475949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:26.475910    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.344453 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.344418    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.436257 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.436215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:27.753142 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.753047    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:01:27.755572 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.755556    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Aug 04 18:01:27.766454 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.766425    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/290108f0-7082-4602-827e-23aa0cc5015c-original-pull-secret\") pod \"global-pull-secret-syncer-hv2mv\" (UID: \"290108f0-7082-4602-827e-23aa0cc5015c\") " pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:01:27.853588 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.853554    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:01:27.853744 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.853633    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:01:27.856015 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.856000    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Aug 04 18:01:27.856086 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.856070    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Aug 04 18:01:27.865920 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.865902    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Aug 04 18:01:27.866233 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.866213    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bea9f6f-a036-4ccb-bc75-5214012b9a70-metrics-certs\") pod \"network-metrics-daemon-hsmdl\" (UID: \"4bea9f6f-a036-4ccb-bc75-5214012b9a70\") " pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:01:27.876361 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.876334    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqfkd\" (UniqueName: \"kubernetes.io/projected/69a6ac8e-d331-4b39-9046-19ad6bead317-kube-api-access-dqfkd\") pod \"network-check-target-wc9ks\" (UID: \"69a6ac8e-d331-4b39-9046-19ad6bead317\") " pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:01:27.979364 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:27.979329    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:28.046986 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.046915    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hp4qc\""
Aug 04 18:01:28.053174 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.053157    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hv2mv"
Aug 04 18:01:28.055281 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.055265    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-hsmdl"
Aug 04 18:01:28.060346 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.060302    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:28.061847 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.061830    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nfg9c\""
Aug 04 18:01:28.069747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.069720    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:01:28.194803 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.194760    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-hsmdl"]
Aug 04 18:01:28.212232 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.212210    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hv2mv"]
Aug 04 18:01:28.215716 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:28.215690    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290108f0_7082_4602_827e_23aa0cc5015c.slice/crio-ca125a1b66913c36273b6ed082bd1a5118fa819a450d291895d3dd0975dc70dc WatchSource:0}: Error finding container ca125a1b66913c36273b6ed082bd1a5118fa819a450d291895d3dd0975dc70dc: Status 404 returned error can't find the container with id ca125a1b66913c36273b6ed082bd1a5118fa819a450d291895d3dd0975dc70dc
Aug 04 18:01:28.229197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.229177    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wc9ks"]
Aug 04 18:01:28.231793 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:28.231761    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a6ac8e_d331_4b39_9046_19ad6bead317.slice/crio-18f10d0069262d29bdffc6690ba18747797aeca27eeb705570bf11d94b7fc825 WatchSource:0}: Error finding container 18f10d0069262d29bdffc6690ba18747797aeca27eeb705570bf11d94b7fc825: Status 404 returned error can't find the container with id 18f10d0069262d29bdffc6690ba18747797aeca27eeb705570bf11d94b7fc825
Aug 04 18:01:28.383268 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.383241    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:28.392953 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.392932    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wc9ks" event={"ID":"69a6ac8e-d331-4b39-9046-19ad6bead317","Type":"ContainerStarted","Data":"18f10d0069262d29bdffc6690ba18747797aeca27eeb705570bf11d94b7fc825"}
Aug 04 18:01:28.393996 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.393975    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hv2mv" event={"ID":"290108f0-7082-4602-827e-23aa0cc5015c","Type":"ContainerStarted","Data":"ca125a1b66913c36273b6ed082bd1a5118fa819a450d291895d3dd0975dc70dc"}
Aug 04 18:01:28.395113 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.395091    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsmdl" event={"ID":"4bea9f6f-a036-4ccb-bc75-5214012b9a70","Type":"ContainerStarted","Data":"e352b4a530af2427a40e16ccb2f1a93a47fdba60e6c62aa7b5d516bd25d43609"}
Aug 04 18:01:28.582253 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.582217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:28.777497 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:28.777420    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.180242 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:29.180204    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.883237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:29.883184    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:29.984915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:29.984875    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:30.389678 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.389638    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:30.401863 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.401776    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsmdl" event={"ID":"4bea9f6f-a036-4ccb-bc75-5214012b9a70","Type":"ContainerStarted","Data":"5b27993fc219f366a90a1cb2977bba343c8440b58f65192978c57251d88c9ed5"}
Aug 04 18:01:30.401863 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.401840    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-hsmdl" event={"ID":"4bea9f6f-a036-4ccb-bc75-5214012b9a70","Type":"ContainerStarted","Data":"85c016ae6693730be4d2d85a67804731db1b5bc4333e9e5bd006e28ccc97cea4"}
Aug 04 18:01:30.420441 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.420340    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-hsmdl" podStartSLOduration=65.474601914 podStartE2EDuration="1m7.420318039s" totalImagesPullingTime="1.945716125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:30.4198068 +0000 UTC m=+67.806497533" watchObservedRunningTime="2026-08-04 18:01:30.420318039 +0000 UTC m=+67.807008775"
Aug 04 18:01:30.579191 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.579147    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:30.773305 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.773226    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:30.878866 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.878820    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:30.879074 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.878888    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:30.879074 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.878932    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:30.879517 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.879490    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/52fba404-4a23-4553-803a-0a1c47fc074d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:30.881598 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.881576    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/52fba404-4a23-4553-803a-0a1c47fc074d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gwkr7\" (UID: \"52fba404-4a23-4553-803a-0a1c47fc074d\") " pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:30.881727 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.881706    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5658ae88-67e4-4545-ae55-5a352c56be47-metrics-tls\") pod \"dns-default-cbzvp\" (UID: \"5658ae88-67e4-4545-ae55-5a352c56be47\") " pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:30.990352 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:30.990313    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:31.027217 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.027133    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vx672\""
Aug 04 18:01:31.035200 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.035170    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gwkr7"
Aug 04 18:01:31.115660 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.115624    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-k2cqb\""
Aug 04 18:01:31.123536 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.123506    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:31.956803 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.956760    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gwkr7"]
Aug 04 18:01:31.961643 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:31.961614    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fba404_4a23_4553_803a_0a1c47fc074d.slice/crio-6ce3f26121f81da5b5e7dd446c8dfe56b1cc605d6ee2c55a25d25e4787890d1b WatchSource:0}: Error finding container 6ce3f26121f81da5b5e7dd446c8dfe56b1cc605d6ee2c55a25d25e4787890d1b: Status 404 returned error can't find the container with id 6ce3f26121f81da5b5e7dd446c8dfe56b1cc605d6ee2c55a25d25e4787890d1b
Aug 04 18:01:31.978372 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:31.978348    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbzvp"]
Aug 04 18:01:31.985170 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:31.984795    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5658ae88_67e4_4545_ae55_5a352c56be47.slice/crio-0460c0a5f3dff3b4d7954793a021681412785e6cba3119d45b78399af1427d95 WatchSource:0}: Error finding container 0460c0a5f3dff3b4d7954793a021681412785e6cba3119d45b78399af1427d95: Status 404 returned error can't find the container with id 0460c0a5f3dff3b4d7954793a021681412785e6cba3119d45b78399af1427d95
Aug 04 18:01:32.411445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:32.411407    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbzvp" event={"ID":"5658ae88-67e4-4545-ae55-5a352c56be47","Type":"ContainerStarted","Data":"0460c0a5f3dff3b4d7954793a021681412785e6cba3119d45b78399af1427d95"}
Aug 04 18:01:32.413424 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:32.413344    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwkr7" event={"ID":"52fba404-4a23-4553-803a-0a1c47fc074d","Type":"ContainerStarted","Data":"a49d61f2c507e4bc9642016a6387d9ba740b3c45eee97994a96fb16a537e5df1"}
Aug 04 18:01:32.413424 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:32.413399    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwkr7" event={"ID":"52fba404-4a23-4553-803a-0a1c47fc074d","Type":"ContainerStarted","Data":"6ce3f26121f81da5b5e7dd446c8dfe56b1cc605d6ee2c55a25d25e4787890d1b"}
Aug 04 18:01:32.414907 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:32.414882    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hv2mv" event={"ID":"290108f0-7082-4602-827e-23aa0cc5015c","Type":"ContainerStarted","Data":"6b9bf488585b5ef68019ebfec3dd0b537af7e6e48b64599d5b780fbee7fc0726"}
Aug 04 18:01:32.432949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:32.432898    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hv2mv" podStartSLOduration=65.793869995 podStartE2EDuration="1m9.432878808s" totalImagesPullingTime="3.639008813s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:32.431372352 +0000 UTC m=+69.818063087" watchObservedRunningTime="2026-08-04 18:01:32.432878808 +0000 UTC m=+69.819569522"
Aug 04 18:01:33.421239 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:33.421205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwkr7" event={"ID":"52fba404-4a23-4553-803a-0a1c47fc074d","Type":"ContainerStarted","Data":"22705eb312c00fd6a0a6a3a1dc79d88ffcc9061fe267d9e8115bfc383cd94c7b"}
Aug 04 18:01:35.430443 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.430400    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbzvp" event={"ID":"5658ae88-67e4-4545-ae55-5a352c56be47","Type":"ContainerStarted","Data":"c7ed5b0650120f8383f9df0277bc423135ef76eac08ebd9d5b8520c7e8d50017"}
Aug 04 18:01:35.430923 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.430468    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbzvp" event={"ID":"5658ae88-67e4-4545-ae55-5a352c56be47","Type":"ContainerStarted","Data":"9fcc28ec51d41d73edaa34439ad721e01877361751f5f6efe471dbef4c3a3c2f"}
Aug 04 18:01:35.430923 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.430497    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:35.432244 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.432213    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wc9ks" event={"ID":"69a6ac8e-d331-4b39-9046-19ad6bead317","Type":"ContainerStarted","Data":"bdd9c1ae1c9268742d0a91385d0e1c4f620b58d371a6d42178379ce4a24738ec"}
Aug 04 18:01:35.432376 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.432287    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:01:35.449138 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.449087    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cbzvp" podStartSLOduration=34.637521159 podStartE2EDuration="37.44906856s" totalImagesPullingTime="2.811547401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:35.44821625 +0000 UTC m=+72.834906984" watchObservedRunningTime="2026-08-04 18:01:35.44906856 +0000 UTC m=+72.835759291"
Aug 04 18:01:35.466152 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.466084    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wc9ks" podStartSLOduration=65.897600458 podStartE2EDuration="1m12.46606658s" totalImagesPullingTime="6.568466122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:00:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:35.465514179 +0000 UTC m=+72.852204914" watchObservedRunningTime="2026-08-04 18:01:35.46606658 +0000 UTC m=+72.852757312"
Aug 04 18:01:35.544255 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.544217    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:35.645380 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.645314    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:35.747304 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.747229    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:35.846415 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:35.846331    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.225925 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:36.225887    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.286523 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:36.286491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:36.623421 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:36.623386    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:37.442434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:37.442404    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gwkr7" event={"ID":"52fba404-4a23-4553-803a-0a1c47fc074d","Type":"ContainerStarted","Data":"f961d485ce4d2080fd5a475fdf53462e09108f2c1ab0ad04fc1fe268815bf002"}
Aug 04 18:01:37.463519 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:37.463464    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gwkr7" podStartSLOduration=18.147589721 podStartE2EDuration="22.463444888s" totalImagesPullingTime="4.315855167s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:01:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:37.462460915 +0000 UTC m=+74.849151649" watchObservedRunningTime="2026-08-04 18:01:37.463444888 +0000 UTC m=+74.850135622"
Aug 04 18:01:39.130078 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.130041    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"]
Aug 04 18:01:39.132560 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.132540    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.135445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.135421    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Aug 04 18:01:39.135575 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.135484    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Aug 04 18:01:39.135700 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.135679    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Aug 04 18:01:39.136173 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.136157    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Aug 04 18:01:39.136241 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.136220    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Aug 04 18:01:39.136503 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.136489    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Aug 04 18:01:39.136944 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.136923    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Aug 04 18:01:39.148259 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.148238    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"]
Aug 04 18:01:39.156098 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.156076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.244531 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244496    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35514649-a720-46c0-9938-2d9eab71400a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.244689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244535    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9br69\" (UniqueName: \"kubernetes.io/projected/35514649-a720-46c0-9938-2d9eab71400a-kube-api-access-9br69\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.244689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244567    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.244689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244630    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.244689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244652    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.244854 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.244704    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.271328 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.271297    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.345291 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.345408 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345294    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.345408 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345318    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.345408 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345338    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35514649-a720-46c0-9938-2d9eab71400a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.345604 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9br69\" (UniqueName: \"kubernetes.io/projected/35514649-a720-46c0-9938-2d9eab71400a-kube-api-access-9br69\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.345604 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.345516    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.346316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.346284    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/35514649-a720-46c0-9938-2d9eab71400a-additional-service-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.347856 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.347836    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-ca\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.348083 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.348067    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.348152 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.348097    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.348152 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.348136    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/35514649-a720-46c0-9938-2d9eab71400a-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.357062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.357017    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9br69\" (UniqueName: \"kubernetes.io/projected/35514649-a720-46c0-9938-2d9eab71400a-kube-api-access-9br69\") pod \"cluster-proxy-proxy-agent-59f68fd48c-qrnvw\" (UID: \"35514649-a720-46c0-9938-2d9eab71400a\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.366268 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.366244    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.451237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.451179    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"
Aug 04 18:01:39.469552 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.469524    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:39.602451 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.602415    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw"]
Aug 04 18:01:39.611590 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:39.611566    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35514649_a720_46c0_9938_2d9eab71400a.slice/crio-d4e10fd4c935e36b42f0f6756a108605442817e54e76f2c7a5caedbb5255777b WatchSource:0}: Error finding container d4e10fd4c935e36b42f0f6756a108605442817e54e76f2c7a5caedbb5255777b: Status 404 returned error can't find the container with id d4e10fd4c935e36b42f0f6756a108605442817e54e76f2c7a5caedbb5255777b
Aug 04 18:01:39.639609 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:39.639578    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:40.455318 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:40.455262    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw" event={"ID":"35514649-a720-46c0-9938-2d9eab71400a","Type":"ContainerStarted","Data":"d4e10fd4c935e36b42f0f6756a108605442817e54e76f2c7a5caedbb5255777b"}
Aug 04 18:01:41.972412 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:41.972376    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:42.068524 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:42.068487    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:42.069243 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:42.069219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9429ca73-a4a0-41a3-a154-5426e63032c9-service-ca-bundle\") pod \"router-default-64dcfc95cb-lqxzk\" (UID: \"9429ca73-a4a0-41a3-a154-5426e63032c9\") " pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:42.236093 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:42.236003    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vw6kw\""
Aug 04 18:01:42.244035 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:42.244005    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:43.033367 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.033339    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-64dcfc95cb-lqxzk"]
Aug 04 18:01:43.036383 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:01:43.036360    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9429ca73_a4a0_41a3_a154_5426e63032c9.slice/crio-cf61756404c5703e6fbb9ccc528c399271e13decd4474bb85066f052e4e1da3a WatchSource:0}: Error finding container cf61756404c5703e6fbb9ccc528c399271e13decd4474bb85066f052e4e1da3a: Status 404 returned error can't find the container with id cf61756404c5703e6fbb9ccc528c399271e13decd4474bb85066f052e4e1da3a
Aug 04 18:01:43.467051 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.466989    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk" event={"ID":"9429ca73-a4a0-41a3-a154-5426e63032c9","Type":"ContainerStarted","Data":"40545e77f68a66744a50d2897a2b0b2462c799335331f53802cc7821a2aebfaa"}
Aug 04 18:01:43.467247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.467060    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk" event={"ID":"9429ca73-a4a0-41a3-a154-5426e63032c9","Type":"ContainerStarted","Data":"cf61756404c5703e6fbb9ccc528c399271e13decd4474bb85066f052e4e1da3a"}
Aug 04 18:01:43.468925 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.468896    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw" event={"ID":"35514649-a720-46c0-9938-2d9eab71400a","Type":"ContainerStarted","Data":"8e959c158849bcd2a63a7a1c99f4cd69417576caa3fe788ba7aa4b7abe00c7ed"}
Aug 04 18:01:43.469062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.468939    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw" event={"ID":"35514649-a720-46c0-9938-2d9eab71400a","Type":"ContainerStarted","Data":"72fa3cfaecbb5d0cf80a22a7ca2ab871ca8265c16678c8d7f64dc586777a4be8"}
Aug 04 18:01:43.469062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.468952    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw" event={"ID":"35514649-a720-46c0-9938-2d9eab71400a","Type":"ContainerStarted","Data":"d620855c1ff37f316652773fb5fd4f5f858faf371fc075055e9f7eaa2228df80"}
Aug 04 18:01:43.491213 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.491168    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk" podStartSLOduration=33.491155169 podStartE2EDuration="33.491155169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:43.489627377 +0000 UTC m=+80.876318110" watchObservedRunningTime="2026-08-04 18:01:43.491155169 +0000 UTC m=+80.877845901"
Aug 04 18:01:43.509531 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:43.509472    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-59f68fd48c-qrnvw" podStartSLOduration=1.181299748 podStartE2EDuration="4.509452787s" totalImagesPullingTime="3.328153039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:01:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:01:43.508837286 +0000 UTC m=+80.895528020" watchObservedRunningTime="2026-08-04 18:01:43.509452787 +0000 UTC m=+80.896143521"
Aug 04 18:01:44.244550 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.244512    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:44.247139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.247114    2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:44.284096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.284062    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.383491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.383454    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:44.472188 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.472157    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:44.473382 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:44.473361    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-64dcfc95cb-lqxzk"
Aug 04 18:01:45.439989 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:45.439961    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cbzvp"
Aug 04 18:01:45.727384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:45.727289    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:45.826866 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:45.826831    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:49.057489 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:49.057459    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:49.130948 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:49.130905    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:50.095434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:50.095393    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:50.184010 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:50.183975    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:50.386688 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:50.386597    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:53.371340 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:53.371299    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:53.471465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:53.471431    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:54.582116 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:54.582079    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:54.644184 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:54.644150    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:54.852298 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:54.852202    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:54.955217 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:54.955181    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:57.560588 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:57.560554    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:57.662587 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:57.662555    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:57.759963 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:57.759928    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:57.866267 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:57.866180    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:58.868785 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:58.868754    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:01:59.552089 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:01:59.552055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.132335 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:01.132297    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.231505 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:01.231449    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.720951 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:01.720904    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.825134 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:01.825096    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:01.915175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:01.915147    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:02.255168 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:02.255128    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:02.551060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:02.550952    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:02.642632 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:02.642600    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:02.723324 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:02.723285    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:03.090436 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:03.090399    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:03.197106 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:03.197072    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:06.420872 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.420837    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-g2qg7"]
Aug 04 18:02:06.424148 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.424130    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.427490 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.427468    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Aug 04 18:02:06.427667 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.427649    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Aug 04 18:02:06.428628 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.428607    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6dtkz\""
Aug 04 18:02:06.428628 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.428623    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Aug 04 18:02:06.428762 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.428607    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Aug 04 18:02:06.428762 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.428645    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Aug 04 18:02:06.428900 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.428874    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Aug 04 18:02:06.439875 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.439853    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wc9ks"
Aug 04 18:02:06.538960 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.538920    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-sys\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.538960 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.538962    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-wtmp\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539072    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt5g7\" (UniqueName: \"kubernetes.io/projected/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-kube-api-access-bt5g7\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539105    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539125    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-root\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539166    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-metrics-client-ca\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539183    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539209    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-textfile\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.539486 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.539243    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-tls\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640009    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-metrics-client-ca\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640073    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640100    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-textfile\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640114    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-tls\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640132    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-sys\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640147    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-wtmp\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640266 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640256    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt5g7\" (UniqueName: \"kubernetes.io/projected/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-kube-api-access-bt5g7\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640295    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-wtmp\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640311    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640340    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-root\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640260    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-sys\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640385    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-root\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640821 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640743    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-metrics-client-ca\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640821 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640777    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-textfile\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.640821 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.640783    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-accelerators-collector-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.642751 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.642731    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.642848 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.642808    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-node-exporter-tls\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.652176 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.652155    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt5g7\" (UniqueName: \"kubernetes.io/projected/3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547-kube-api-access-bt5g7\") pod \"node-exporter-g2qg7\" (UID: \"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547\") " pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:06.733690 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:06.733609    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-g2qg7"
Aug 04 18:02:07.545527 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:07.545494    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2qg7" event={"ID":"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547","Type":"ContainerStarted","Data":"5e59d8972336825c9ec06e554ede91b49713e927eadd53a340ebee16b9e0d7ec"}
Aug 04 18:02:07.936973 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:07.936935    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:08.038973 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:08.038942    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:09.375440 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.375401    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:09.488195 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.488161    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:09.554563 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.554526    2554 generic.go:356] "Generic (PLEG): container finished" podID="3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547" containerID="ec152da72584b09a699d6903c474eaf436caf9b4fc164cc46cb59101de5cbae6" exitCode=0
Aug 04 18:02:09.554774 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.554578    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2qg7" event={"ID":"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547","Type":"ContainerDied","Data":"ec152da72584b09a699d6903c474eaf436caf9b4fc164cc46cb59101de5cbae6"}
Aug 04 18:02:09.700134 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.700100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:09.813548 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:09.813517    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:10.289396 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:10.289363    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:10.395107 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:10.395073    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:10.560346 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:10.560316    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2qg7" event={"ID":"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547","Type":"ContainerStarted","Data":"934ed969b1622c3320063711f59ac4f0f3d3f65e9fc37282dccbeb4516dd6bf5"}
Aug 04 18:02:10.560504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:10.560366    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-g2qg7" event={"ID":"3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547","Type":"ContainerStarted","Data":"34c423e5cfa93833b8ba5e61ac32a13825ed821feef7075158daba738f54029a"}
Aug 04 18:02:10.582522 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:10.582473    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-g2qg7" podStartSLOduration=2.1943906970000002 podStartE2EDuration="4.582453419s" totalImagesPullingTime="2.274061831s" totalInitContainerRuntime="114.000891ms" isStatefulPod=true podCreationTimestamp="2026-08-04 18:02:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:02:10.582278335 +0000 UTC m=+107.968969068" watchObservedRunningTime="2026-08-04 18:02:10.582453419 +0000 UTC m=+107.969144152"
Aug 04 18:02:11.112465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:11.112435    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:11.218488 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:11.218455    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:12.741077 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:12.741038    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:12.831808 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:12.831768    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:13.872251 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:13.872203    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:13.976133 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:13.976081    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:14.206211 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:14.206126    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:14.310347 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:14.310315    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:16.133242 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:16.133203    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:16.252075 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:16.252021    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:16.349060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:16.349008    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:16.858558 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:16.858517    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:16.961441 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:16.961404    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:22.067183 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:22.067148    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:22.142392 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:22.142353    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:29.029308 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:29.029269    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:29.128567 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:29.128537    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:29.295388 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:29.295277    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:30.628712 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:30.628680    2554 generic.go:356] "Generic (PLEG): container finished" podID="754f774e-53e8-4b98-bea2-8c64eeea8f28" containerID="8c89244d2bd8074ff5f45643266d3619aa9fc58a4ef9ee8963797e3c14319a73" exitCode=0
Aug 04 18:02:30.629183 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:30.628726    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v" event={"ID":"754f774e-53e8-4b98-bea2-8c64eeea8f28","Type":"ContainerDied","Data":"8c89244d2bd8074ff5f45643266d3619aa9fc58a4ef9ee8963797e3c14319a73"}
Aug 04 18:02:30.629183 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:30.629060    2554 scope.go:117] "RemoveContainer" containerID="8c89244d2bd8074ff5f45643266d3619aa9fc58a4ef9ee8963797e3c14319a73"
Aug 04 18:02:31.633805 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:31.633771    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7878f4b88c-x789v" event={"ID":"754f774e-53e8-4b98-bea2-8c64eeea8f28","Type":"ContainerStarted","Data":"f97789834e5d05671f5484743c85a02431d05176203355249e8c6f8b5552518d"}
Aug 04 18:02:37.105359 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:37.105325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:02:37.247157 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:02:37.247118    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:11.233364 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:11.233322    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:22.542528 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:22.542443    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:22.618598 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:22.618542    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:34.587405 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:34.587353    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:34.648820 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:34.648784    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:35.804214 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:35.804177    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:35.883087 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:35.883055    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:37.076719 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:37.076676    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:37.173959 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:37.173923    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:46.284499 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:46.284463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:46.382288 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:46.382255    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:46.458009 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:46.457979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:03:58.190207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:03:58.190174    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:15.391207 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:04:15.391172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:15.459445 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:04:15.459413    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:45.202763 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:04:45.202669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:48.282715 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:04:48.282672    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:04:48.347816 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:04:48.347784    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:22.959144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:22.959119    2554 kubelet.go:1648] "Image garbage collection succeeded"
Aug 04 18:05:34.552350 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:34.552314    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:34.644853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:34.644821    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:34.708504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:34.708474    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:05:35.577509 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.577475    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"]
Aug 04 18:05:35.580633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.580617    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.583230 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.583209    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Aug 04 18:05:35.584083 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.584069    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-nmcvv\""
Aug 04 18:05:35.584138 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.584070    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Aug 04 18:05:35.591380 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.591358    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"]
Aug 04 18:05:35.655106 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.655080    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zphw\" (UniqueName: \"kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.655208 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.655126    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.655208 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.655158    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.756284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.756257    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.756384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.756296    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.756384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.756348    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zphw\" (UniqueName: \"kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.756629 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.756609    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.756705 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.756657    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.766198 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.766174    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zphw\" (UniqueName: \"kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:35.890007 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:35.889957    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:36.017111 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:36.017083    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"]
Aug 04 18:05:36.020135 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:05:36.020101    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod167cc38a_209d_4807_9d6b_feaa986cf9fc.slice/crio-23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e WatchSource:0}: Error finding container 23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e: Status 404 returned error can't find the container with id 23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e
Aug 04 18:05:36.021986 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:36.021971    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:05:36.241536 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:36.241462    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb" event={"ID":"167cc38a-209d-4807-9d6b-feaa986cf9fc","Type":"ContainerStarted","Data":"23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e"}
Aug 04 18:05:43.265272 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:43.265235    2554 generic.go:356] "Generic (PLEG): container finished" podID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerID="221918d104f51bc0b667bcb3976f59f15e1bce837fdb854c6476f6f047051565" exitCode=0
Aug 04 18:05:43.265686 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:43.265326    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb" event={"ID":"167cc38a-209d-4807-9d6b-feaa986cf9fc","Type":"ContainerDied","Data":"221918d104f51bc0b667bcb3976f59f15e1bce837fdb854c6476f6f047051565"}
Aug 04 18:05:47.281566 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:47.281526    2554 generic.go:356] "Generic (PLEG): container finished" podID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerID="49e253aec016c870ca4051c366a278707e213a3045035beff7040a34cfba9480" exitCode=0
Aug 04 18:05:47.281970 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:47.281584    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb" event={"ID":"167cc38a-209d-4807-9d6b-feaa986cf9fc","Type":"ContainerDied","Data":"49e253aec016c870ca4051c366a278707e213a3045035beff7040a34cfba9480"}
Aug 04 18:05:54.310197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:54.310162    2554 generic.go:356] "Generic (PLEG): container finished" podID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerID="a07dd8a3b16ce797614f40dde632ff3c25b103fd90eceb3182c95c1e23672321" exitCode=0
Aug 04 18:05:54.310569 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:54.310248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb" event={"ID":"167cc38a-209d-4807-9d6b-feaa986cf9fc","Type":"ContainerDied","Data":"a07dd8a3b16ce797614f40dde632ff3c25b103fd90eceb3182c95c1e23672321"}
Aug 04 18:05:55.429137 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.429114    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:55.605397 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.605313    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util\") pod \"167cc38a-209d-4807-9d6b-feaa986cf9fc\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") "
Aug 04 18:05:55.605397 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.605357    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4zphw\" (UniqueName: \"kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw\") pod \"167cc38a-209d-4807-9d6b-feaa986cf9fc\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") "
Aug 04 18:05:55.605615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.605401    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle\") pod \"167cc38a-209d-4807-9d6b-feaa986cf9fc\" (UID: \"167cc38a-209d-4807-9d6b-feaa986cf9fc\") "
Aug 04 18:05:55.606054 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.606010    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle" (OuterVolumeSpecName: "bundle") pod "167cc38a-209d-4807-9d6b-feaa986cf9fc" (UID: "167cc38a-209d-4807-9d6b-feaa986cf9fc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:05:55.607700 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.607671    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw" (OuterVolumeSpecName: "kube-api-access-4zphw") pod "167cc38a-209d-4807-9d6b-feaa986cf9fc" (UID: "167cc38a-209d-4807-9d6b-feaa986cf9fc"). InnerVolumeSpecName "kube-api-access-4zphw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:05:55.610310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.610285    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util" (OuterVolumeSpecName: "util") pod "167cc38a-209d-4807-9d6b-feaa986cf9fc" (UID: "167cc38a-209d-4807-9d6b-feaa986cf9fc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:05:55.706652 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.706622    2554 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-bundle\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:05:55.706652 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.706646    2554 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/167cc38a-209d-4807-9d6b-feaa986cf9fc-util\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:05:55.706652 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:55.706655    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zphw\" (UniqueName: \"kubernetes.io/projected/167cc38a-209d-4807-9d6b-feaa986cf9fc-kube-api-access-4zphw\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:05:56.318956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:56.318926    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb"
Aug 04 18:05:56.319136 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:56.318926    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5l28rb" event={"ID":"167cc38a-209d-4807-9d6b-feaa986cf9fc","Type":"ContainerDied","Data":"23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e"}
Aug 04 18:05:56.319136 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:05:56.319056    2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23730b309d99559646845ab5710b18d5d083069cf60a608c63ccc26abb53282e"
Aug 04 18:06:01.887957 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:01.887921    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:04.847525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:04.847493    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:07.253645 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.253604    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:07.556698 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.556624    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:07.563104 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563082    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-zhsnv"]
Aug 04 18:06:07.563308 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563297    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="util"
Aug 04 18:06:07.563354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563309    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="util"
Aug 04 18:06:07.563354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563318    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="pull"
Aug 04 18:06:07.563354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563323    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="pull"
Aug 04 18:06:07.563354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563336    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="extract"
Aug 04 18:06:07.563354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563342    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="extract"
Aug 04 18:06:07.563500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.563382    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="167cc38a-209d-4807-9d6b-feaa986cf9fc" containerName="extract"
Aug 04 18:06:07.566206 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.566192    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.571224 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.571199    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Aug 04 18:06:07.571384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.571266    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Aug 04 18:06:07.571384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.571361    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Aug 04 18:06:07.571503 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.571400    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-tkd5p\""
Aug 04 18:06:07.572096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.572008    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Aug 04 18:06:07.572330 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.572311    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Aug 04 18:06:07.575997 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.575978    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zhsnv"]
Aug 04 18:06:07.686883 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.686848    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee80cff0-a1ea-4f3d-915a-b459c53015be-cabundle0\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.687074 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.686894    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vv9sv\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-kube-api-access-vv9sv\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.687074 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.686918    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.787782 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.787755    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee80cff0-a1ea-4f3d-915a-b459c53015be-cabundle0\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.787870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.787808    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vv9sv\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-kube-api-access-vv9sv\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.787870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.787831    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.787946 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:07.787921    2554 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Aug 04 18:06:07.787946 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:07.787935    2554 secret.go:281] references non-existent secret key: ca.crt
Aug 04 18:06:07.787946 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:07.787943    2554 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Aug 04 18:06:07.788087 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:07.787954    2554 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zhsnv: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Aug 04 18:06:07.788087 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:07.788009    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates podName:ee80cff0-a1ea-4f3d-915a-b459c53015be nodeName:}" failed. No retries permitted until 2026-08-04 18:06:08.287994855 +0000 UTC m=+345.674685566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates") pod "keda-operator-77f648d694-zhsnv" (UID: "ee80cff0-a1ea-4f3d-915a-b459c53015be") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Aug 04 18:06:07.788397 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.788377    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/ee80cff0-a1ea-4f3d-915a-b459c53015be-cabundle0\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.797112 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.797095    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vv9sv\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-kube-api-access-vv9sv\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:07.874190 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.874152    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:07.997175 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:07.997144    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:08.291524 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.291443    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:08.291899 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:08.291583    2554 secret.go:281] references non-existent secret key: ca.crt
Aug 04 18:06:08.291899 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:08.291603    2554 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Aug 04 18:06:08.291899 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:08.291612    2554 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-zhsnv: references non-existent secret key: ca.crt
Aug 04 18:06:08.291899 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:06:08.291675    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates podName:ee80cff0-a1ea-4f3d-915a-b459c53015be nodeName:}" failed. No retries permitted until 2026-08-04 18:06:09.291660691 +0000 UTC m=+346.678351402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates") pod "keda-operator-77f648d694-zhsnv" (UID: "ee80cff0-a1ea-4f3d-915a-b459c53015be") : references non-existent secret key: ca.crt
Aug 04 18:06:08.607691 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.607661    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-c6c58"]
Aug 04 18:06:08.610883 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.610866    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.613258 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.613242    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Aug 04 18:06:08.623979 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.623959    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-c6c58"]
Aug 04 18:06:08.795752 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.795721    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7rn5\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-kube-api-access-n7rn5\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.795918 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.795770    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-certificates\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.896224 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.896148    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7rn5\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-kube-api-access-n7rn5\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.896224 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.896202    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-certificates\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.898903 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.898883    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-certificates\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.908372 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.908341    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7rn5\" (UniqueName: \"kubernetes.io/projected/1d71f4ab-bf27-44af-8576-1fe663182742-kube-api-access-n7rn5\") pod \"keda-admission-56f9798bb4-c6c58\" (UID: \"1d71f4ab-bf27-44af-8576-1fe663182742\") " pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:08.920118 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:08.920097    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:09.045713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.045682    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-c6c58"]
Aug 04 18:06:09.048693 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:06:09.048664    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d71f4ab_bf27_44af_8576_1fe663182742.slice/crio-a4cd4f2f7d53454b4ab6775432a580629fdd897892e9801da215c6e5771561d5 WatchSource:0}: Error finding container a4cd4f2f7d53454b4ab6775432a580629fdd897892e9801da215c6e5771561d5: Status 404 returned error can't find the container with id a4cd4f2f7d53454b4ab6775432a580629fdd897892e9801da215c6e5771561d5
Aug 04 18:06:09.299384 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.299290    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:09.301856 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.301824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/ee80cff0-a1ea-4f3d-915a-b459c53015be-certificates\") pod \"keda-operator-77f648d694-zhsnv\" (UID: \"ee80cff0-a1ea-4f3d-915a-b459c53015be\") " pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:09.365553 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.365523    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-c6c58" event={"ID":"1d71f4ab-bf27-44af-8576-1fe663182742","Type":"ContainerStarted","Data":"a4cd4f2f7d53454b4ab6775432a580629fdd897892e9801da215c6e5771561d5"}
Aug 04 18:06:09.375812 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.375783    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:09.505506 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.505471    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-zhsnv"]
Aug 04 18:06:09.507997 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:06:09.507971    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee80cff0_a1ea_4f3d_915a_b459c53015be.slice/crio-f3b906efbd2d326b0b7d201b2d833ab05f0357efa6c6f9dfc23a6da6f76d4c83 WatchSource:0}: Error finding container f3b906efbd2d326b0b7d201b2d833ab05f0357efa6c6f9dfc23a6da6f76d4c83: Status 404 returned error can't find the container with id f3b906efbd2d326b0b7d201b2d833ab05f0357efa6c6f9dfc23a6da6f76d4c83
Aug 04 18:06:09.551199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.551137    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:09.647205 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:09.647169    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:06:10.370516 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:10.370478    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zhsnv" event={"ID":"ee80cff0-a1ea-4f3d-915a-b459c53015be","Type":"ContainerStarted","Data":"f3b906efbd2d326b0b7d201b2d833ab05f0357efa6c6f9dfc23a6da6f76d4c83"}
Aug 04 18:06:11.376775 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:11.376741    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-c6c58" event={"ID":"1d71f4ab-bf27-44af-8576-1fe663182742","Type":"ContainerStarted","Data":"6253fbefac316459cf0d2a5ef174a044b4ff4eca162704f5417e42850da2a552"}
Aug 04 18:06:11.377176 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:11.376846    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:11.395955 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:11.395892    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-c6c58" podStartSLOduration=1.136077152 podStartE2EDuration="3.395870876s" totalImagesPullingTime="2.259793724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:11.394974498 +0000 UTC m=+348.781665235" watchObservedRunningTime="2026-08-04 18:06:11.395870876 +0000 UTC m=+348.782561610"
Aug 04 18:06:13.386426 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:13.386387    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-zhsnv" event={"ID":"ee80cff0-a1ea-4f3d-915a-b459c53015be","Type":"ContainerStarted","Data":"e03244defe2f7a056d5537bb3e8c287e875a0da1f4ad64469a75367a5a99a498"}
Aug 04 18:06:13.386807 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:13.386457    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:06:32.383944 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:32.383904    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-c6c58"
Aug 04 18:06:32.400952 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:32.400900    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-zhsnv" podStartSLOduration=21.959267892 podStartE2EDuration="25.400884211s" totalImagesPullingTime="3.441616319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:06:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:06:13.414110889 +0000 UTC m=+350.800801639" watchObservedRunningTime="2026-08-04 18:06:32.400884211 +0000 UTC m=+369.787574945"
Aug 04 18:06:34.392101 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:06:34.392072    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-zhsnv"
Aug 04 18:07:00.315649 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:00.315612    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:00.379351 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:00.379314    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:25.717331 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.717282    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:25.783378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.783344    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:07:25.785580 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.785559    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:25.788254 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.788235    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-qmh2t\""
Aug 04 18:07:25.788353 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.788270    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Aug 04 18:07:25.789166 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.789148    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Aug 04 18:07:25.789259 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.789181    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Aug 04 18:07:25.799044 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.799001    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:07:25.903179 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.903148    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:25.903358 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.903190    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5k2p\" (UniqueName: \"kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:25.942144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:25.942105    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.004080 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.003961    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.004080 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.004012    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5k2p\" (UniqueName: \"kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.004299 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:07:26.004124    2554 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Aug 04 18:07:26.004299 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:07:26.004185    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert podName:5d7c7a7d-4c54-4012-a03c-292d456efc11 nodeName:}" failed. No retries permitted until 2026-08-04 18:07:26.504167646 +0000 UTC m=+423.890858357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert") pod "kserve-controller-manager-5f9d7f69b6-4nnkx" (UID: "5d7c7a7d-4c54-4012-a03c-292d456efc11") : secret "kserve-webhook-server-cert" not found
Aug 04 18:07:26.020011 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.019983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5k2p\" (UniqueName: \"kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.071787 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.071744    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.214281 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.214244    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.286382 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.286290    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.507697 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.507655    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.510217 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.510197    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") pod \"kserve-controller-manager-5f9d7f69b6-4nnkx\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.526005 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.525975    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.603917 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.603893    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:26.695101 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.695076    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:26.832690 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:26.832660    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:07:26.835838 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:07:26.835810    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7c7a7d_4c54_4012_a03c_292d456efc11.slice/crio-f0d8f9e095b3d070779ce1a618a12b9363fac09b6f081f2a4b782bd6a52a0126 WatchSource:0}: Error finding container f0d8f9e095b3d070779ce1a618a12b9363fac09b6f081f2a4b782bd6a52a0126: Status 404 returned error can't find the container with id f0d8f9e095b3d070779ce1a618a12b9363fac09b6f081f2a4b782bd6a52a0126
Aug 04 18:07:27.639128 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:27.639072    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" event={"ID":"5d7c7a7d-4c54-4012-a03c-292d456efc11","Type":"ContainerStarted","Data":"f0d8f9e095b3d070779ce1a618a12b9363fac09b6f081f2a4b782bd6a52a0126"}
Aug 04 18:07:29.648148 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:29.648114    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" event={"ID":"5d7c7a7d-4c54-4012-a03c-292d456efc11","Type":"ContainerStarted","Data":"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"}
Aug 04 18:07:29.648519 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:29.648250    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:07:29.665471 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:29.665434    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" podStartSLOduration=2.080690322 podStartE2EDuration="4.66542347s" totalImagesPullingTime="2.584733148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:07:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:07:29.664289939 +0000 UTC m=+427.050980683" watchObservedRunningTime="2026-08-04 18:07:29.66542347 +0000 UTC m=+427.052114202"
Aug 04 18:07:42.781212 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:42.781172    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:44.827979 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:44.827899    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:07:44.928902 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:07:44.928877    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:00.656349 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:00.656318    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:08:04.247648 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:04.247603    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:04.348355 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:04.348324    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:09.757854 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.757818    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:08:09.758396 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.758049    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" podUID="5d7c7a7d-4c54-4012-a03c-292d456efc11" containerName="manager" containerID="cri-o://db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f" gracePeriod=10
Aug 04 18:08:09.786827 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.786794    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"]
Aug 04 18:08:09.789822 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.789805    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.798569 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.798535    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"]
Aug 04 18:08:09.831712 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.831688    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-cert\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.831854 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.831762    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddj9\" (UniqueName: \"kubernetes.io/projected/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-kube-api-access-nddj9\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.932517 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.932487    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-cert\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.932644 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.932545    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddj9\" (UniqueName: \"kubernetes.io/projected/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-kube-api-access-nddj9\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.935021 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.934998    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-cert\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.941277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.941255    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddj9\" (UniqueName: \"kubernetes.io/projected/e4276eda-9d2c-472f-b753-b0b7ccfc1ced-kube-api-access-nddj9\") pod \"kserve-controller-manager-5f9d7f69b6-ssbn8\" (UID: \"e4276eda-9d2c-472f-b753-b0b7ccfc1ced\") " pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:09.998983 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:09.998953    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:08:10.035060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.034970    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") pod \"5d7c7a7d-4c54-4012-a03c-292d456efc11\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") "
Aug 04 18:08:10.035060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.035015    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5k2p\" (UniqueName: \"kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p\") pod \"5d7c7a7d-4c54-4012-a03c-292d456efc11\" (UID: \"5d7c7a7d-4c54-4012-a03c-292d456efc11\") "
Aug 04 18:08:10.037276 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.037231    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert" (OuterVolumeSpecName: "cert") pod "5d7c7a7d-4c54-4012-a03c-292d456efc11" (UID: "5d7c7a7d-4c54-4012-a03c-292d456efc11"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:08:10.037539 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.037519    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p" (OuterVolumeSpecName: "kube-api-access-c5k2p") pod "5d7c7a7d-4c54-4012-a03c-292d456efc11" (UID: "5d7c7a7d-4c54-4012-a03c-292d456efc11"). InnerVolumeSpecName "kube-api-access-c5k2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:08:10.136247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.136210    2554 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5d7c7a7d-4c54-4012-a03c-292d456efc11-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:08:10.136247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.136240    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5k2p\" (UniqueName: \"kubernetes.io/projected/5d7c7a7d-4c54-4012-a03c-292d456efc11-kube-api-access-c5k2p\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:08:10.139181 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.139160    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:10.263910 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.263852    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"]
Aug 04 18:08:10.266385 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:08:10.266359    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4276eda_9d2c_472f_b753_b0b7ccfc1ced.slice/crio-73c3cddfa7dc1c3d2a69cb822cb06ecf2a3c90ae6f9b14212214d85832eb1c20 WatchSource:0}: Error finding container 73c3cddfa7dc1c3d2a69cb822cb06ecf2a3c90ae6f9b14212214d85832eb1c20: Status 404 returned error can't find the container with id 73c3cddfa7dc1c3d2a69cb822cb06ecf2a3c90ae6f9b14212214d85832eb1c20
Aug 04 18:08:10.796241 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.796203    2554 generic.go:356] "Generic (PLEG): container finished" podID="5d7c7a7d-4c54-4012-a03c-292d456efc11" containerID="db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f" exitCode=0
Aug 04 18:08:10.796634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.796246    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"
Aug 04 18:08:10.796634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.796285    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" event={"ID":"5d7c7a7d-4c54-4012-a03c-292d456efc11","Type":"ContainerDied","Data":"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"}
Aug 04 18:08:10.796634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.796340    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-4nnkx" event={"ID":"5d7c7a7d-4c54-4012-a03c-292d456efc11","Type":"ContainerDied","Data":"f0d8f9e095b3d070779ce1a618a12b9363fac09b6f081f2a4b782bd6a52a0126"}
Aug 04 18:08:10.796634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.796384    2554 scope.go:117] "RemoveContainer" containerID="db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"
Aug 04 18:08:10.797798 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.797461    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8" event={"ID":"e4276eda-9d2c-472f-b753-b0b7ccfc1ced","Type":"ContainerStarted","Data":"73c3cddfa7dc1c3d2a69cb822cb06ecf2a3c90ae6f9b14212214d85832eb1c20"}
Aug 04 18:08:10.805271 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.805253    2554 scope.go:117] "RemoveContainer" containerID="db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"
Aug 04 18:08:10.805517 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:08:10.805496    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f\": container with ID starting with db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f not found: ID does not exist" containerID="db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"
Aug 04 18:08:10.805644 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.805530    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f"} err="failed to get container status \"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f\": rpc error: code = NotFound desc = could not find container \"db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f\": container with ID starting with db6ccb772849c903226cf3125a115b7c24371c632fa8b5068ba84565a2db3d8f not found: ID does not exist"
Aug 04 18:08:10.819668 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.819646    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:08:10.823706 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:10.823681    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-5f9d7f69b6-4nnkx"]
Aug 04 18:08:11.136047 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:11.135998    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d7c7a7d-4c54-4012-a03c-292d456efc11" path="/var/lib/kubelet/pods/5d7c7a7d-4c54-4012-a03c-292d456efc11/volumes"
Aug 04 18:08:11.803199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:11.803162    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8" event={"ID":"e4276eda-9d2c-472f-b753-b0b7ccfc1ced","Type":"ContainerStarted","Data":"8d7767adbd667b1acd15e1759e8f2e59ea8b0c07132549887c6b68ed07389ad2"}
Aug 04 18:08:11.803628 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:11.803285    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:11.825916 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:11.825870    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8" podStartSLOduration=2.173751313 podStartE2EDuration="2.825855289s" totalImagesPullingTime="652.103976ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:08:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:08:11.82489758 +0000 UTC m=+469.211588312" watchObservedRunningTime="2026-08-04 18:08:11.825855289 +0000 UTC m=+469.212546022"
Aug 04 18:08:12.082742 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:12.082712    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:42.811341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:42.811306    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-5f9d7f69b6-ssbn8"
Aug 04 18:08:43.912414 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:43.912368    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:43.982226 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:43.982190    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.259048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:44.258945    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.356396 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:44.356364    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.456918 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:44.456883    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.623225 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:44.623187    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:08:44.682132 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:08:44.682093    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:14.101748 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:09:14.101669    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:09:40.846443 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:09:40.846399    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:10.195407 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:10.195374    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:19.455802 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:19.455759    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:19.558927 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:19.558889    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:30.936530 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:30.936491    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:31.047159 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:31.047118    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:31.190954 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:31.190874    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:10:31.288863 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:10:31.288834    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:20.661109 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.661011    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:20.676121 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.676089    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:12:20.676425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.676409    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d7c7a7d-4c54-4012-a03c-292d456efc11" containerName="manager"
Aug 04 18:12:20.676494 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.676429    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d7c7a7d-4c54-4012-a03c-292d456efc11" containerName="manager"
Aug 04 18:12:20.676549 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.676518    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d7c7a7d-4c54-4012-a03c-292d456efc11" containerName="manager"
Aug 04 18:12:20.679405 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.679386    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.681857 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.681834    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:12:20.681955 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.681863    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:12:20.681955 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.681908    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:12:20.682967 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.682945    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Aug 04 18:12:20.683053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.682952    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:12:20.683053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.683046    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:12:20.683145 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.683053    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Aug 04 18:12:20.691504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.691487    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:12:20.771045 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.770992    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.771045 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.771047    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.771282 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.771071    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74vjv\" (UniqueName: \"kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.771282 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.771147    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.771282 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.771195    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.872506 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.872472    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.872506 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.872507    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.872720 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.872536    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.872720 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.872555    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.872720 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.872584    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74vjv\" (UniqueName: \"kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.873053 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.873004    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.873228 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.873210    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.873295 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.873278    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.875003 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.874983    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.883578 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.883560    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74vjv\" (UniqueName: \"kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv\") pod \"isvc-canary-rollback-predictor-7f8946b8f4-tzljt\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:20.989908 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:20.989832    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:21.119969 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:21.119808    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:12:21.122443 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:12:21.122409    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bb133d_38f6_40c3_8d88_c33f06f45be2.slice/crio-24fe30ae9eb630aa2c2d1d60111c84ed29f2d44a593ae25b449220b83b66d290 WatchSource:0}: Error finding container 24fe30ae9eb630aa2c2d1d60111c84ed29f2d44a593ae25b449220b83b66d290: Status 404 returned error can't find the container with id 24fe30ae9eb630aa2c2d1d60111c84ed29f2d44a593ae25b449220b83b66d290
Aug 04 18:12:21.124529 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:21.124510    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:12:21.649297 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:21.649258    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerStarted","Data":"24fe30ae9eb630aa2c2d1d60111c84ed29f2d44a593ae25b449220b83b66d290"}
Aug 04 18:12:24.755401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:24.755380    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:12:25.664557 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:25.664520    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerStarted","Data":"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"}
Aug 04 18:12:25.774236 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:25.774189    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:25.880766 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:25.880695    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:25.971811 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:25.971762    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:26.043304 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:26.043264    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:26.319375 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:26.319341    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:28.679600 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:28.679565    2554 generic.go:356] "Generic (PLEG): container finished" podID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerID="fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d" exitCode=0
Aug 04 18:12:28.679969 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:28.679641    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerDied","Data":"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"}
Aug 04 18:12:36.534708 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:36.534651    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:36.637010 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:36.636972    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:42.745341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:42.745263    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerStarted","Data":"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"}
Aug 04 18:12:44.756325 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:44.756291    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerStarted","Data":"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"}
Aug 04 18:12:44.756638 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:44.756516    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:44.756689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:44.756679    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:44.757809 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:44.757780    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:12:44.775495 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:44.775456    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podStartSLOduration=1.578761573 podStartE2EDuration="24.775443441s" totalImagesPullingTime="19.538116689s" totalInitContainerRuntime="3.658565179s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:12:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:12:44.774445925 +0000 UTC m=+742.161136657" watchObservedRunningTime="2026-08-04 18:12:44.775443441 +0000 UTC m=+742.162134173"
Aug 04 18:12:45.760985 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:45.760947    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:12:46.764779 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:46.764740    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:12:47.519707 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:47.519676    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:47.625068 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:47.625005    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:47.725977 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:47.725948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:12:51.769913 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:51.769887    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:12:51.770383 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:51.770355    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:12:57.753241 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:12:57.753210    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:01.770508 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:01.770466    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:11.770446 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:11.770409    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:21.770952 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:21.770911    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:31.771194 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:31.771150    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:41.771279 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:41.771239    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:49.118857 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:49.118774    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:13:51.770539 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:51.770504    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.19:8080: connect: connection refused"
Aug 04 18:13:56.132317 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:13:56.132283    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:14:00.712014 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:00.711978    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:14:00.712572 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:00.712334    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container" containerID="cri-o://3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6" gracePeriod=30
Aug 04 18:14:00.712572 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:00.712362    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kube-rbac-proxy" containerID="cri-o://f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99" gracePeriod=30
Aug 04 18:14:00.740610 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:00.740585    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:01.029020 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:01.028943    2554 generic.go:356] "Generic (PLEG): container finished" podID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerID="f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99" exitCode=2
Aug 04 18:14:01.029020 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:01.029002    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerDied","Data":"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"}
Aug 04 18:14:01.765316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:01.765273    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.19:8643/healthz\": dial tcp 10.134.0.19:8643: connect: connection refused"
Aug 04 18:14:04.954789 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:04.954764    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:14:05.044409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044339    2554 generic.go:356] "Generic (PLEG): container finished" podID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerID="3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6" exitCode=0
Aug 04 18:14:05.044409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044376    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerDied","Data":"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"}
Aug 04 18:14:05.044409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044403    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044432    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt" event={"ID":"58bb133d-38f6-40c3-8d88-c33f06f45be2","Type":"ContainerDied","Data":"24fe30ae9eb630aa2c2d1d60111c84ed29f2d44a593ae25b449220b83b66d290"}
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044460    2554 scope.go:117] "RemoveContainer" containerID="f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044479    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location\") pod \"58bb133d-38f6-40c3-8d88-c33f06f45be2\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") "
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044504    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74vjv\" (UniqueName: \"kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv\") pod \"58bb133d-38f6-40c3-8d88-c33f06f45be2\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") "
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044547    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config\") pod \"58bb133d-38f6-40c3-8d88-c33f06f45be2\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") "
Aug 04 18:14:05.044599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044584    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert\") pod \"58bb133d-38f6-40c3-8d88-c33f06f45be2\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") "
Aug 04 18:14:05.044871 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044631    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls\") pod \"58bb133d-38f6-40c3-8d88-c33f06f45be2\" (UID: \"58bb133d-38f6-40c3-8d88-c33f06f45be2\") "
Aug 04 18:14:05.044947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.044858    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "58bb133d-38f6-40c3-8d88-c33f06f45be2" (UID: "58bb133d-38f6-40c3-8d88-c33f06f45be2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:14:05.045059 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.045010    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "58bb133d-38f6-40c3-8d88-c33f06f45be2" (UID: "58bb133d-38f6-40c3-8d88-c33f06f45be2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:05.045059 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.045021    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "58bb133d-38f6-40c3-8d88-c33f06f45be2" (UID: "58bb133d-38f6-40c3-8d88-c33f06f45be2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:14:05.046924 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.046890    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv" (OuterVolumeSpecName: "kube-api-access-74vjv") pod "58bb133d-38f6-40c3-8d88-c33f06f45be2" (UID: "58bb133d-38f6-40c3-8d88-c33f06f45be2"). InnerVolumeSpecName "kube-api-access-74vjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:14:05.047009 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.046972    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "58bb133d-38f6-40c3-8d88-c33f06f45be2" (UID: "58bb133d-38f6-40c3-8d88-c33f06f45be2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:14:05.060851 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.060811    2554 scope.go:117] "RemoveContainer" containerID="3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"
Aug 04 18:14:05.067883 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.067865    2554 scope.go:117] "RemoveContainer" containerID="fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"
Aug 04 18:14:05.074402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.074385    2554 scope.go:117] "RemoveContainer" containerID="f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"
Aug 04 18:14:05.074646 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:14:05.074630    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99\": container with ID starting with f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99 not found: ID does not exist" containerID="f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"
Aug 04 18:14:05.074689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.074654    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99"} err="failed to get container status \"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99\": rpc error: code = NotFound desc = could not find container \"f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99\": container with ID starting with f175b867869f5f99203fdac4bf0cbf5d638d1401f43831187c5424ff2431dc99 not found: ID does not exist"
Aug 04 18:14:05.074689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.074670    2554 scope.go:117] "RemoveContainer" containerID="3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"
Aug 04 18:14:05.074905 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:14:05.074889    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6\": container with ID starting with 3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6 not found: ID does not exist" containerID="3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"
Aug 04 18:14:05.074949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.074908    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6"} err="failed to get container status \"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6\": rpc error: code = NotFound desc = could not find container \"3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6\": container with ID starting with 3081a150fc29de35270237028bb8fc3505bc170f291219c518c84cc8376266b6 not found: ID does not exist"
Aug 04 18:14:05.074949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.074921    2554 scope.go:117] "RemoveContainer" containerID="fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"
Aug 04 18:14:05.075118 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:14:05.075103    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d\": container with ID starting with fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d not found: ID does not exist" containerID="fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"
Aug 04 18:14:05.075163 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.075120    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d"} err="failed to get container status \"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d\": rpc error: code = NotFound desc = could not find container \"fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d\": container with ID starting with fb0f7b61c2949e80e57eccfa404defb3bd8f2e1ea934af2da3eb622d8b65f33d not found: ID does not exist"
Aug 04 18:14:05.145605 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.145587    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.145674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.145607    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/58bb133d-38f6-40c3-8d88-c33f06f45be2-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.145674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.145617    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/58bb133d-38f6-40c3-8d88-c33f06f45be2-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.145674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.145625    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/58bb133d-38f6-40c3-8d88-c33f06f45be2-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.145674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.145633    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74vjv\" (UniqueName: \"kubernetes.io/projected/58bb133d-38f6-40c3-8d88-c33f06f45be2-kube-api-access-74vjv\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:14:05.362465 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.362442    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:14:05.366729 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.366702    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f8946b8f4-tzljt"]
Aug 04 18:14:05.840877 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.840844    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:05.908436 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908406    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:14:05.908674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908662    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kube-rbac-proxy"
Aug 04 18:14:05.908713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908677    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kube-rbac-proxy"
Aug 04 18:14:05.908713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908686    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container"
Aug 04 18:14:05.908713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908693    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container"
Aug 04 18:14:05.908713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908706    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="storage-initializer"
Aug 04 18:14:05.908713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908712    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="storage-initializer"
Aug 04 18:14:05.908859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908779    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kserve-container"
Aug 04 18:14:05.908859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.908789    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" containerName="kube-rbac-proxy"
Aug 04 18:14:05.912221 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.912200    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:05.914872 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.914852    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:14:05.915713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.915697    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:14:05.915835 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.915703    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:14:05.915996 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.915976    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Aug 04 18:14:05.916158 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.916001    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:14:05.916158 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.916095    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:14:05.916249 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.916209    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Aug 04 18:14:05.921357 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.921335    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:14:05.923482 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.923443    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:05.926062 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.926003    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-v2-predictor-serving-cert\""
Aug 04 18:14:05.927529 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.927511    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:14:05.938885 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.938859    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:05.945461 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:05.945440    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:14:06.015438 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.015400    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:06.051404 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051373    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj427\" (UniqueName: \"kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.051576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051408    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.051576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051474    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6mlh\" (UniqueName: \"kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.051576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.051576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051535    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.051744 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051595    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.051744 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051630    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.051744 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051657    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.051744 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051713    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.051944 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.051747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.152871 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152785    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj427\" (UniqueName: \"kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.152871 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152826    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.152871 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152864    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6mlh\" (UniqueName: \"kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153164 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152890    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153164 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152916    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.153164 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.152957    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.153164 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153087    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153164 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153128    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153187    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153227    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.153401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153393    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.153753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153695    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.153753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.153753 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153771    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.154077 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153872    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.154077 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.153813    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.155625 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.155599    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.155728 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.155704    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.161853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.161821    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6mlh\" (UniqueName: \"kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh\") pod \"isvc-canary-stop-predictor-57599f75db-4zxgc\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.162359 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.162341    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj427\" (UniqueName: \"kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427\") pod \"isvc-canary-stop-v2-predictor-55768c9995-k976p\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") " pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.222521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.222482    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:06.233360 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.233329    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:06.359564 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.359524    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:14:06.362299 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:14:06.362270    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0439a194_20cf_428a_bd6d_e70d122d45b1.slice/crio-e08cc4e1820c08640107560b876f1d4ccf0ac7d65674a94923a647a883bf028e WatchSource:0}: Error finding container e08cc4e1820c08640107560b876f1d4ccf0ac7d65674a94923a647a883bf028e: Status 404 returned error can't find the container with id e08cc4e1820c08640107560b876f1d4ccf0ac7d65674a94923a647a883bf028e
Aug 04 18:14:06.393115 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.393094    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:14:06.395468 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:14:06.395443    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd29b6b16_695c_4c5e_aae6_2251777de35c.slice/crio-2864a0ccd3f58e978b58df1e45a85b372b24e7a3e9510d982a5818bf84bd03e8 WatchSource:0}: Error finding container 2864a0ccd3f58e978b58df1e45a85b372b24e7a3e9510d982a5818bf84bd03e8: Status 404 returned error can't find the container with id 2864a0ccd3f58e978b58df1e45a85b372b24e7a3e9510d982a5818bf84bd03e8
Aug 04 18:14:06.934322 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:06.934286    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:07.055442 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:07.055411    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerStarted","Data":"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"}
Aug 04 18:14:07.055878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:07.055473    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerStarted","Data":"e08cc4e1820c08640107560b876f1d4ccf0ac7d65674a94923a647a883bf028e"}
Aug 04 18:14:07.056945 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:07.056918    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerStarted","Data":"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"}
Aug 04 18:14:07.057063 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:07.056965    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerStarted","Data":"2864a0ccd3f58e978b58df1e45a85b372b24e7a3e9510d982a5818bf84bd03e8"}
Aug 04 18:14:07.139107 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:07.139079    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bb133d-38f6-40c3-8d88-c33f06f45be2" path="/var/lib/kubelet/pods/58bb133d-38f6-40c3-8d88-c33f06f45be2/volumes"
Aug 04 18:14:10.071264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:10.071229    2554 generic.go:356] "Generic (PLEG): container finished" podID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerID="f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7" exitCode=0
Aug 04 18:14:10.071676 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:10.071297    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerDied","Data":"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"}
Aug 04 18:14:10.072826 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:10.072806    2554 generic.go:356] "Generic (PLEG): container finished" podID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerID="249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b" exitCode=0
Aug 04 18:14:10.072896 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:10.072842    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerDied","Data":"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"}
Aug 04 18:14:11.078269 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.078235    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerStarted","Data":"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"}
Aug 04 18:14:11.078657 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.078311    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerStarted","Data":"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"}
Aug 04 18:14:11.078657 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.078528    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:11.080247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.080225    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerStarted","Data":"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"}
Aug 04 18:14:11.080352 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.080277    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerStarted","Data":"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"}
Aug 04 18:14:11.080531 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.080518    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:11.080575 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.080556    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:11.082143 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.082115    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:11.096901 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.096860    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podStartSLOduration=2.528845955 podStartE2EDuration="6.096848048s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.568002093s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:11.095399653 +0000 UTC m=+828.482090385" watchObservedRunningTime="2026-08-04 18:14:11.096848048 +0000 UTC m=+828.483538780"
Aug 04 18:14:11.113397 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:11.113356    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podStartSLOduration=2.50996642 podStartE2EDuration="6.113343526s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.603377106s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:14:11.112073914 +0000 UTC m=+828.498764650" watchObservedRunningTime="2026-08-04 18:14:11.113343526 +0000 UTC m=+828.500034331"
Aug 04 18:14:12.085901 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:12.085846    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:12.086331 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:12.085847    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:12.087437 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:12.087409    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:13.089946 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:13.089901    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:15.983146 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:15.983108    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:16.086154 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:16.086116    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:16.157747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:16.157713    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:16.254322 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:16.254237    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:16.333109 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:16.333073    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:16.534335 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:16.534253    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:14:17.090977 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:17.090945    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:14:17.091533 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:17.091507    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:18.095008 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:18.094981    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:14:18.095557 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:18.095530    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:27.091471 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:27.091429    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:28.095803 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:28.095755    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:37.091475 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:37.091437    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:38.095941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:38.095903    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:47.092359 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:47.092318    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:48.096344 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:48.096302    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:14:57.091661 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:57.091617    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:14:58.095427 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:14:58.095387    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:15:07.091682 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:07.091643    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:15:08.096000 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:08.095957    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:15:17.091785 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:17.091683    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Aug 04 18:15:18.095608 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:18.095569    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:15:22.132829 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:22.132798    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:15:27.092174 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:27.092143    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:15:30.875599 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.875561    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:15:30.876112 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.875845    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" containerID="cri-o://eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad" gracePeriod=30
Aug 04 18:15:30.876112 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.875893    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kube-rbac-proxy" containerID="cri-o://6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639" gracePeriod=30
Aug 04 18:15:30.888184 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.888143    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:30.929364 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.929330    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:15:30.929548 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.929528    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container" containerID="cri-o://0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab" gracePeriod=30
Aug 04 18:15:30.929670 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.929611    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kube-rbac-proxy" containerID="cri-o://2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68" gracePeriod=30
Aug 04 18:15:30.979052 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:30.979010    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:31.372945 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:31.372914    2554 generic.go:356] "Generic (PLEG): container finished" podID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerID="6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639" exitCode=2
Aug 04 18:15:31.373160 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:31.372993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerDied","Data":"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"}
Aug 04 18:15:31.374987 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:31.374968    2554 generic.go:356] "Generic (PLEG): container finished" podID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerID="2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68" exitCode=2
Aug 04 18:15:31.375102 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:31.374995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerDied","Data":"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"}
Aug 04 18:15:32.086278 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:32.086232    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Aug 04 18:15:32.132294 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:32.132253    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Aug 04 18:15:33.090615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:33.090563    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Aug 04 18:15:35.527824 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.527798    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:15:35.572564 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.572544    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:15:35.628402 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628376    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config\") pod \"0439a194-20cf-428a-bd6d-e70d122d45b1\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") "
Aug 04 18:15:35.628515 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628425    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c6mlh\" (UniqueName: \"kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh\") pod \"0439a194-20cf-428a-bd6d-e70d122d45b1\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") "
Aug 04 18:15:35.628515 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628455    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location\") pod \"0439a194-20cf-428a-bd6d-e70d122d45b1\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") "
Aug 04 18:15:35.628515 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628483    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls\") pod \"0439a194-20cf-428a-bd6d-e70d122d45b1\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") "
Aug 04 18:15:35.628642 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628524    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert\") pod \"0439a194-20cf-428a-bd6d-e70d122d45b1\" (UID: \"0439a194-20cf-428a-bd6d-e70d122d45b1\") "
Aug 04 18:15:35.628809 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628783    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0439a194-20cf-428a-bd6d-e70d122d45b1" (UID: "0439a194-20cf-428a-bd6d-e70d122d45b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:35.628852 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628752    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0439a194-20cf-428a-bd6d-e70d122d45b1" (UID: "0439a194-20cf-428a-bd6d-e70d122d45b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:35.628915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.628896    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0439a194-20cf-428a-bd6d-e70d122d45b1" (UID: "0439a194-20cf-428a-bd6d-e70d122d45b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:35.630586 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.630563    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0439a194-20cf-428a-bd6d-e70d122d45b1" (UID: "0439a194-20cf-428a-bd6d-e70d122d45b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:35.630655 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.630639    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh" (OuterVolumeSpecName: "kube-api-access-c6mlh") pod "0439a194-20cf-428a-bd6d-e70d122d45b1" (UID: "0439a194-20cf-428a-bd6d-e70d122d45b1"). InnerVolumeSpecName "kube-api-access-c6mlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:35.729565 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729535    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert\") pod \"d29b6b16-695c-4c5e-aae6-2251777de35c\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") "
Aug 04 18:15:35.729716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729604    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tj427\" (UniqueName: \"kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427\") pod \"d29b6b16-695c-4c5e-aae6-2251777de35c\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") "
Aug 04 18:15:35.729716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729634    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls\") pod \"d29b6b16-695c-4c5e-aae6-2251777de35c\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") "
Aug 04 18:15:35.729716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729674    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location\") pod \"d29b6b16-695c-4c5e-aae6-2251777de35c\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") "
Aug 04 18:15:35.729716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729700    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config\") pod \"d29b6b16-695c-4c5e-aae6-2251777de35c\" (UID: \"d29b6b16-695c-4c5e-aae6-2251777de35c\") "
Aug 04 18:15:35.729913 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729871    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6mlh\" (UniqueName: \"kubernetes.io/projected/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-api-access-c6mlh\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.729913 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729889    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0439a194-20cf-428a-bd6d-e70d122d45b1-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.729913 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729905    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0439a194-20cf-428a-bd6d-e70d122d45b1-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.730069 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729922    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.730069 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729938    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0439a194-20cf-428a-bd6d-e70d122d45b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.730069 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.729979    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d29b6b16-695c-4c5e-aae6-2251777de35c" (UID: "d29b6b16-695c-4c5e-aae6-2251777de35c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:15:35.730069 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.730008    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d29b6b16-695c-4c5e-aae6-2251777de35c" (UID: "d29b6b16-695c-4c5e-aae6-2251777de35c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:35.730247 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.730082    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d29b6b16-695c-4c5e-aae6-2251777de35c" (UID: "d29b6b16-695c-4c5e-aae6-2251777de35c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:15:35.731793 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.731764    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d29b6b16-695c-4c5e-aae6-2251777de35c" (UID: "d29b6b16-695c-4c5e-aae6-2251777de35c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:15:35.731857 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.731802    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427" (OuterVolumeSpecName: "kube-api-access-tj427") pod "d29b6b16-695c-4c5e-aae6-2251777de35c" (UID: "d29b6b16-695c-4c5e-aae6-2251777de35c"). InnerVolumeSpecName "kube-api-access-tj427". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:15:35.830277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.830206    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tj427\" (UniqueName: \"kubernetes.io/projected/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-api-access-tj427\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.830277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.830232    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d29b6b16-695c-4c5e-aae6-2251777de35c-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.830277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.830246    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d29b6b16-695c-4c5e-aae6-2251777de35c-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.830277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.830257    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:35.830277 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:35.830269    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d29b6b16-695c-4c5e-aae6-2251777de35c-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:15:36.395843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.395808    2554 generic.go:356] "Generic (PLEG): container finished" podID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerID="eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad" exitCode=0
Aug 04 18:15:36.396001 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.395859    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"
Aug 04 18:15:36.396001 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.395905    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerDied","Data":"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"}
Aug 04 18:15:36.396001 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.395979    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc" event={"ID":"0439a194-20cf-428a-bd6d-e70d122d45b1","Type":"ContainerDied","Data":"e08cc4e1820c08640107560b876f1d4ccf0ac7d65674a94923a647a883bf028e"}
Aug 04 18:15:36.396205 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.396017    2554 scope.go:117] "RemoveContainer" containerID="6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"
Aug 04 18:15:36.398041 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.398007    2554 generic.go:356] "Generic (PLEG): container finished" podID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerID="0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab" exitCode=0
Aug 04 18:15:36.398153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.398053    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerDied","Data":"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"}
Aug 04 18:15:36.398153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.398093    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"
Aug 04 18:15:36.398153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.398115    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p" event={"ID":"d29b6b16-695c-4c5e-aae6-2251777de35c","Type":"ContainerDied","Data":"2864a0ccd3f58e978b58df1e45a85b372b24e7a3e9510d982a5818bf84bd03e8"}
Aug 04 18:15:36.405096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.405077    2554 scope.go:117] "RemoveContainer" containerID="eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"
Aug 04 18:15:36.412359 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.412343    2554 scope.go:117] "RemoveContainer" containerID="f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"
Aug 04 18:15:36.419683 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.419668    2554 scope.go:117] "RemoveContainer" containerID="6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"
Aug 04 18:15:36.419901 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.419875    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639\": container with ID starting with 6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639 not found: ID does not exist" containerID="6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"
Aug 04 18:15:36.419968 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.419903    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639"} err="failed to get container status \"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639\": rpc error: code = NotFound desc = could not find container \"6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639\": container with ID starting with 6734cb1c4885d92e48d60f76aa97e3b5be0818aa0a1de34e25fe4b371b8d3639 not found: ID does not exist"
Aug 04 18:15:36.419968 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.419920    2554 scope.go:117] "RemoveContainer" containerID="eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"
Aug 04 18:15:36.420147 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.420132    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad\": container with ID starting with eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad not found: ID does not exist" containerID="eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"
Aug 04 18:15:36.420189 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.420153    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad"} err="failed to get container status \"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad\": rpc error: code = NotFound desc = could not find container \"eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad\": container with ID starting with eaf769fa235330909d492ef534476ca0715aa6d60382b70a32d1599efccd67ad not found: ID does not exist"
Aug 04 18:15:36.420189 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.420168    2554 scope.go:117] "RemoveContainer" containerID="f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"
Aug 04 18:15:36.420385 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.420369    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7\": container with ID starting with f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7 not found: ID does not exist" containerID="f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"
Aug 04 18:15:36.420430 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.420387    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7"} err="failed to get container status \"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7\": rpc error: code = NotFound desc = could not find container \"f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7\": container with ID starting with f20cafee4be4881f9480b7cf08dfa67a4fcfcf7aeaca2bd56ce7344a9e8e5ae7 not found: ID does not exist"
Aug 04 18:15:36.420430 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.420398    2554 scope.go:117] "RemoveContainer" containerID="2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"
Aug 04 18:15:36.427692 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.427672    2554 scope.go:117] "RemoveContainer" containerID="0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"
Aug 04 18:15:36.428958 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.428940    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:15:36.435449 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.435429    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-v2-predictor-55768c9995-k976p"]
Aug 04 18:15:36.435515 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.435462    2554 scope.go:117] "RemoveContainer" containerID="249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"
Aug 04 18:15:36.445109 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.445087    2554 scope.go:117] "RemoveContainer" containerID="2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"
Aug 04 18:15:36.445397 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.445375    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68\": container with ID starting with 2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68 not found: ID does not exist" containerID="2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"
Aug 04 18:15:36.445524 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.445406    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68"} err="failed to get container status \"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68\": rpc error: code = NotFound desc = could not find container \"2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68\": container with ID starting with 2390fa4d3df8023e9431bf77b93b69bfd3c5eefe16bd2ffcda7f63c0c908fd68 not found: ID does not exist"
Aug 04 18:15:36.445524 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.445429    2554 scope.go:117] "RemoveContainer" containerID="0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"
Aug 04 18:15:36.445738 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.445721    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab\": container with ID starting with 0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab not found: ID does not exist" containerID="0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"
Aug 04 18:15:36.445799 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.445745    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab"} err="failed to get container status \"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab\": rpc error: code = NotFound desc = could not find container \"0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab\": container with ID starting with 0463287a6eb5dcb827f999206bf958c5c99c7d795d2ffdc26133ccbf172dbcab not found: ID does not exist"
Aug 04 18:15:36.445799 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.445763    2554 scope.go:117] "RemoveContainer" containerID="249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"
Aug 04 18:15:36.446001 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:15:36.445981    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b\": container with ID starting with 249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b not found: ID does not exist" containerID="249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"
Aug 04 18:15:36.446060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.446007    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b"} err="failed to get container status \"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b\": rpc error: code = NotFound desc = could not find container \"249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b\": container with ID starting with 249237f194a92a6d60cbc7af8e2d09bf62e71980eb988500ab214e287712940b not found: ID does not exist"
Aug 04 18:15:36.452900 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.452868    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:15:36.457180 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:36.457160    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-57599f75db-4zxgc"]
Aug 04 18:15:37.136825 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:37.136787    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" path="/var/lib/kubelet/pods/0439a194-20cf-428a-bd6d-e70d122d45b1/volumes"
Aug 04 18:15:37.137462 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:37.137441    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" path="/var/lib/kubelet/pods/d29b6b16-695c-4c5e-aae6-2251777de35c/volumes"
Aug 04 18:15:41.162329 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.162296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:41.208057 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208012    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:15:41.208283 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208269    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container"
Aug 04 18:15:41.208283 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208284    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208292    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="storage-initializer"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208299    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="storage-initializer"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208306    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208312    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208322    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208327    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208338    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="storage-initializer"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208343    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="storage-initializer"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208349    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container"
Aug 04 18:15:41.208386 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208354    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container"
Aug 04 18:15:41.208747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208399    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kserve-container"
Aug 04 18:15:41.208747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208407    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208413    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d29b6b16-695c-4c5e-aae6-2251777de35c" containerName="kube-rbac-proxy"
Aug 04 18:15:41.208747 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.208419    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0439a194-20cf-428a-bd6d-e70d122d45b1" containerName="kserve-container"
Aug 04 18:15:41.213120 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.213101    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.215664 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.215644    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:15:41.215809 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.215649    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:15:41.215809 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.215697    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:15:41.216014 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.215996    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:15:41.216104 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.216002    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-beeb9e-predictor-serving-cert\""
Aug 04 18:15:41.216500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.216479    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:15:41.216583 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.216503    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-beeb9e-kube-rbac-proxy-sar-config\""
Aug 04 18:15:41.233486 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.233452    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:15:41.329301 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.329269    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:41.365737 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.365707    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.365870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.365747    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.365870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.365781    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mqw\" (UniqueName: \"kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.365870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.365831    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.365969 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.365890    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466479 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466393    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mqw\" (UniqueName: \"kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466479 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466442    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466479 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466461    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466724 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466494    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466724 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466521    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.466854 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.466830    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.467169 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.467149    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.467267 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.467221    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.468966 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.468949    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.475554 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.475535    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mqw\" (UniqueName: \"kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw\") pod \"isvc-primary-beeb9e-predictor-6f5c477646-qk7f7\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") " pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.523011 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.522987    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:41.659755 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:41.659718    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:15:41.663633 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:15:41.663610    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc594e62_ff31_4c0c_8df0_b12f4366dedd.slice/crio-b607ee80eee8a70e8ae435d82960fa9ad907a6b1b1836ddc7b40a2c759eb7a62 WatchSource:0}: Error finding container b607ee80eee8a70e8ae435d82960fa9ad907a6b1b1836ddc7b40a2c759eb7a62: Status 404 returned error can't find the container with id b607ee80eee8a70e8ae435d82960fa9ad907a6b1b1836ddc7b40a2c759eb7a62
Aug 04 18:15:42.421004 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:42.420968    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerStarted","Data":"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"}
Aug 04 18:15:42.421471 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:42.421057    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerStarted","Data":"b607ee80eee8a70e8ae435d82960fa9ad907a6b1b1836ddc7b40a2c759eb7a62"}
Aug 04 18:15:45.433409 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:45.433371    2554 generic.go:356] "Generic (PLEG): container finished" podID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerID="93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915" exitCode=0
Aug 04 18:15:45.433756 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:45.433418    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerDied","Data":"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"}
Aug 04 18:15:46.440240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:46.440205    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerStarted","Data":"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"}
Aug 04 18:15:46.442547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:46.440264    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerStarted","Data":"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"}
Aug 04 18:15:46.442547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:46.440451    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:46.465922 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:46.465882    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podStartSLOduration=1.9199892969999999 podStartE2EDuration="5.465869306s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.545880009s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:15:46.464390196 +0000 UTC m=+923.851080928" watchObservedRunningTime="2026-08-04 18:15:46.465869306 +0000 UTC m=+923.852560038"
Aug 04 18:15:47.444448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:47.444412    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:47.445775 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:47.445749    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:15:48.451420 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:48.449505    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:15:53.453950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:53.453920    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:15:53.454340 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:53.454316    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:15:56.255363 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:56.255322    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:15:56.342219 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:15:56.342187    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:03.454568 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:03.454528    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:06.345075 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.345014    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:06.519146 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.519092    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:06.532633 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.532601    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:16:06.535892 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.535872    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.538201 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.538180    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-fe589-predictor-serving-cert\""
Aug 04 18:16:06.538201 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.538202    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-fe589-kube-rbac-proxy-sar-config\""
Aug 04 18:16:06.542310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.542097    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.542310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.542176    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.542310 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.542215    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9pwf\" (UniqueName: \"kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.547982 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.547961    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:16:06.594695 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.594660    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:06.642844 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.642768    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.642844 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.642835    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.643079 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.642876    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9pwf\" (UniqueName: \"kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.643079 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:16:06.642972    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-serving-cert: secret "message-dumper-raw-fe589-predictor-serving-cert" not found
Aug 04 18:16:06.643079 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:16:06.643061    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls podName:9ea408dc-c522-4d80-897d-b5e63cf6689b nodeName:}" failed. No retries permitted until 2026-08-04 18:16:07.143022354 +0000 UTC m=+944.529713069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls") pod "message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" (UID: "9ea408dc-c522-4d80-897d-b5e63cf6689b") : secret "message-dumper-raw-fe589-predictor-serving-cert" not found
Aug 04 18:16:06.643474 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.643455    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.653149 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.653122    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9pwf\" (UniqueName: \"kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:06.672000 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.671968    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:06.898302 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:06.898215    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:07.146305 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:07.146268    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:07.148769 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:07.148707    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") pod \"message-dumper-raw-fe589-predictor-6fbb78db75-45mt9\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") " pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:07.445578 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:07.445495    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:07.582362 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:07.582185    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:16:07.584908 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:16:07.584879    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea408dc_c522_4d80_897d_b5e63cf6689b.slice/crio-62cb9a480969945958d997049baf0c7f41cf057e2d9f1ee25ebd156bbed02f8e WatchSource:0}: Error finding container 62cb9a480969945958d997049baf0c7f41cf057e2d9f1ee25ebd156bbed02f8e: Status 404 returned error can't find the container with id 62cb9a480969945958d997049baf0c7f41cf057e2d9f1ee25ebd156bbed02f8e
Aug 04 18:16:08.523044 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:08.522993    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerStarted","Data":"62cb9a480969945958d997049baf0c7f41cf057e2d9f1ee25ebd156bbed02f8e"}
Aug 04 18:16:09.528199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:09.528168    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerStarted","Data":"37d7f923cee3da5c94e2c5ae70beda604c4f339f81a34a31b794f93e2c736b92"}
Aug 04 18:16:09.528621 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:09.528221    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerStarted","Data":"f8976ac7383d424b3e1cf4a4fd118cf18c93c1a24ca7aab84a7d16c45464dc9c"}
Aug 04 18:16:09.528621 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:09.528310    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:09.548311 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:09.548268    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" podStartSLOduration=2.358595751 podStartE2EDuration="3.548252189s" totalImagesPullingTime="1.189656438s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:16:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:16:09.546287437 +0000 UTC m=+946.932978171" watchObservedRunningTime="2026-08-04 18:16:09.548252189 +0000 UTC m=+946.934942923"
Aug 04 18:16:10.532320 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:10.532283    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:10.534197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:10.534166    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:13.454691 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:13.454653    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:17.545260 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:17.545233    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:16:23.454913 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:23.454868    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:26.541321 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:26.541274    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:26.704990 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:26.704948    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:16:33.455263 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:33.455224    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:43.454455 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:43.454367    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:53.455096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:53.455060    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Aug 04 18:16:59.135880 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:16:59.135847    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:17:16.348624 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.348572    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:16.388020 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.387985    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:16.391557 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.391536    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.394390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.394342    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-beeb9e-predictor-serving-cert\""
Aug 04 18:17:16.394593 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.394398    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-beeb9e\""
Aug 04 18:17:16.394676 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.394471    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-beeb9e-dockercfg-fkq6j\""
Aug 04 18:17:16.394676 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.394656    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-beeb9e-kube-rbac-proxy-sar-config\""
Aug 04 18:17:16.405099 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.405075    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:16.550885 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.550857    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf22v\" (UniqueName: \"kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.551048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.550895    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.551048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.550927    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.551048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.550967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.551048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.551020    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651401 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651385    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651416    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651470    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651499    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf22v\" (UniqueName: \"kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.651547 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:16.651514    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-serving-cert: secret "isvc-secondary-beeb9e-predictor-serving-cert" not found
Aug 04 18:17:16.651701 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:16.651583    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls podName:052b6519-46a5-40b4-a873-2bf1f03d8f4a nodeName:}" failed. No retries permitted until 2026-08-04 18:17:17.151562681 +0000 UTC m=+1014.538253392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls") pod "isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a") : secret "isvc-secondary-beeb9e-predictor-serving-cert" not found
Aug 04 18:17:16.651825 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.651806    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.652097 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.652082    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.652144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.652129    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:16.661569 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:16.661544    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf22v\" (UniqueName: \"kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:17.122043 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.121985    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:17.155674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.155647    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:17.158276 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.158246    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") pod \"isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") " pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:17.302246 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.302213    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:17.432615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.432593    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:17.435105 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:17:17.435077    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052b6519_46a5_40b4_a873_2bf1f03d8f4a.slice/crio-1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6 WatchSource:0}: Error finding container 1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6: Status 404 returned error can't find the container with id 1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6
Aug 04 18:17:17.782151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.782067    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerStarted","Data":"eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"}
Aug 04 18:17:17.782151 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:17.782131    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerStarted","Data":"1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6"}
Aug 04 18:17:21.797707 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:21.797680    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/0.log"
Aug 04 18:17:21.798099 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:21.797718    2554 generic.go:356] "Generic (PLEG): container finished" podID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerID="eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41" exitCode=1
Aug 04 18:17:21.798099 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:21.797794    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerDied","Data":"eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"}
Aug 04 18:17:22.803387 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:22.803363    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/0.log"
Aug 04 18:17:22.803776 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:22.803426    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerStarted","Data":"1c2e17d7bb3df7841fe2836ae7da6a6d029c36e0e920547ec0c893a0034d38cf"}
Aug 04 18:17:24.812353 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.812327    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/1.log"
Aug 04 18:17:24.812793 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.812718    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/0.log"
Aug 04 18:17:24.812793 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.812756    2554 generic.go:356] "Generic (PLEG): container finished" podID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerID="1c2e17d7bb3df7841fe2836ae7da6a6d029c36e0e920547ec0c893a0034d38cf" exitCode=1
Aug 04 18:17:24.812908 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.812830    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerDied","Data":"1c2e17d7bb3df7841fe2836ae7da6a6d029c36e0e920547ec0c893a0034d38cf"}
Aug 04 18:17:24.812908 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.812896    2554 scope.go:117] "RemoveContainer" containerID="eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"
Aug 04 18:17:24.813294 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.813274    2554 scope.go:117] "RemoveContainer" containerID="eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"
Aug 04 18:17:24.823748 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:24.823719    2554 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_kserve-ci-e2e-test(052b6519-46a5-40b4-a873-2bf1f03d8f4a)\"" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a"
Aug 04 18:17:24.824014 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:24.823986    2554 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_kserve-ci-e2e-test_052b6519-46a5-40b4-a873-2bf1f03d8f4a_0 in pod sandbox 1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6 from index: no such id: 'eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41'" containerID="eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"
Aug 04 18:17:24.824109 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:24.824041    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_kserve-ci-e2e-test_052b6519-46a5-40b4-a873-2bf1f03d8f4a_0 in pod sandbox 1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6 from index: no such id: 'eda0117b6e07542bfbb022539c12cb9f237750d536a613f74c9ca868d7c03e41'"
Aug 04 18:17:25.818521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:25.818491    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/1.log"
Aug 04 18:17:32.412533 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.412498    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:32.459498 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.459463    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:32.477745 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.477724    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:17:32.478092 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.478003    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container" containerID="cri-o://84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729" gracePeriod=30
Aug 04 18:17:32.478436 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.478382    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kube-rbac-proxy" containerID="cri-o://eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032" gracePeriod=30
Aug 04 18:17:32.597927 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.597909    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/1.log"
Aug 04 18:17:32.600236 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.600218    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:32.677768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.677705    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") pod \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") "
Aug 04 18:17:32.677768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.677736    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nf22v\" (UniqueName: \"kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v\") pod \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") "
Aug 04 18:17:32.677910 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.677781    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert\") pod \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") "
Aug 04 18:17:32.677910 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.677898    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config\") pod \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") "
Aug 04 18:17:32.677997 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.677936    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location\") pod \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\" (UID: \"052b6519-46a5-40b4-a873-2bf1f03d8f4a\") "
Aug 04 18:17:32.678125 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.678103    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "052b6519-46a5-40b4-a873-2bf1f03d8f4a" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:32.678263 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.678235    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "052b6519-46a5-40b4-a873-2bf1f03d8f4a" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:32.678263 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.678247    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "052b6519-46a5-40b4-a873-2bf1f03d8f4a" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:32.679849 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.679830    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "052b6519-46a5-40b4-a873-2bf1f03d8f4a" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:32.679950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.679931    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v" (OuterVolumeSpecName: "kube-api-access-nf22v") pod "052b6519-46a5-40b4-a873-2bf1f03d8f4a" (UID: "052b6519-46a5-40b4-a873-2bf1f03d8f4a"). InnerVolumeSpecName "kube-api-access-nf22v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:32.779453 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.779413    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.779453 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.779448    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.779453 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.779458    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.779634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.779468    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/052b6519-46a5-40b4-a873-2bf1f03d8f4a-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.779634 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.779478    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf22v\" (UniqueName: \"kubernetes.io/projected/052b6519-46a5-40b4-a873-2bf1f03d8f4a-kube-api-access-nf22v\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:32.845763 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.845743    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw_052b6519-46a5-40b4-a873-2bf1f03d8f4a/storage-initializer/1.log"
Aug 04 18:17:32.845866 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.845836    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"
Aug 04 18:17:32.845866 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.845847    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw" event={"ID":"052b6519-46a5-40b4-a873-2bf1f03d8f4a","Type":"ContainerDied","Data":"1b1ba50ad6aaac7c00af60c4edab81c8c8aa13a6cf72289eb357aa43d5ae7dd6"}
Aug 04 18:17:32.845961 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.845918    2554 scope.go:117] "RemoveContainer" containerID="1c2e17d7bb3df7841fe2836ae7da6a6d029c36e0e920547ec0c893a0034d38cf"
Aug 04 18:17:32.848230 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.848204    2554 generic.go:356] "Generic (PLEG): container finished" podID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerID="eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032" exitCode=2
Aug 04 18:17:32.848326 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.848276    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerDied","Data":"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"}
Aug 04 18:17:32.893421 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.893396    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:32.901959 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:32.901937    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-beeb9e-predictor-68d8cb56cc-vt2zw"]
Aug 04 18:17:33.135718 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:33.135685    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" path="/var/lib/kubelet/pods/052b6519-46a5-40b4-a873-2bf1f03d8f4a/volumes"
Aug 04 18:17:33.450003 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:33.449915    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.22:8643/healthz\": dial tcp 10.134.0.22:8643: connect: connection refused"
Aug 04 18:17:33.977663 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:33.977626    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:37.119773 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.119747    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:17:37.213170 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213088    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6mqw\" (UniqueName: \"kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw\") pod \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") "
Aug 04 18:17:37.213170 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213147    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls\") pod \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") "
Aug 04 18:17:37.213385 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213182    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config\") pod \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") "
Aug 04 18:17:37.213385 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213215    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location\") pod \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") "
Aug 04 18:17:37.213385 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213242    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert\") pod \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\" (UID: \"fc594e62-ff31-4c0c-8df0-b12f4366dedd\") "
Aug 04 18:17:37.213565 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213534    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fc594e62-ff31-4c0c-8df0-b12f4366dedd" (UID: "fc594e62-ff31-4c0c-8df0-b12f4366dedd"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:37.213643 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213601    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fc594e62-ff31-4c0c-8df0-b12f4366dedd" (UID: "fc594e62-ff31-4c0c-8df0-b12f4366dedd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:37.213643 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.213623    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fc594e62-ff31-4c0c-8df0-b12f4366dedd" (UID: "fc594e62-ff31-4c0c-8df0-b12f4366dedd"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:37.215361 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.215334    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw" (OuterVolumeSpecName: "kube-api-access-r6mqw") pod "fc594e62-ff31-4c0c-8df0-b12f4366dedd" (UID: "fc594e62-ff31-4c0c-8df0-b12f4366dedd"). InnerVolumeSpecName "kube-api-access-r6mqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:37.215474 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.215402    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fc594e62-ff31-4c0c-8df0-b12f4366dedd" (UID: "fc594e62-ff31-4c0c-8df0-b12f4366dedd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:37.314672 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.314636    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:37.314672 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.314665    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:37.314672 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.314676    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fc594e62-ff31-4c0c-8df0-b12f4366dedd-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:37.314861 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.314684    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mqw\" (UniqueName: \"kubernetes.io/projected/fc594e62-ff31-4c0c-8df0-b12f4366dedd-kube-api-access-r6mqw\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:37.314861 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.314692    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fc594e62-ff31-4c0c-8df0-b12f4366dedd-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:37.691108 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.691076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:37.742522 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742486    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:37.742793 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742779    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742795    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742807    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kube-rbac-proxy"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742813    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kube-rbac-proxy"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742820    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742825    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742834    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="storage-initializer"
Aug 04 18:17:37.742843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742839    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="storage-initializer"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742850    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742855    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742898    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742905    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kube-rbac-proxy"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742912    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="052b6519-46a5-40b4-a873-2bf1f03d8f4a" containerName="storage-initializer"
Aug 04 18:17:37.743144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.742919    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerName="kserve-container"
Aug 04 18:17:37.746019 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.746003    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.752664 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.752632    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-7f063c\""
Aug 04 18:17:37.752773 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.752632    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-7f063c-dockercfg-b29nk\""
Aug 04 18:17:37.753118 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.753102    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7f063c-predictor-serving-cert\""
Aug 04 18:17:37.753203 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.753182    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-7f063c-kube-rbac-proxy-sar-config\""
Aug 04 18:17:37.765719 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.765697    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:37.819057 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.819016    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.819057 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.819059    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljzk\" (UniqueName: \"kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.819271 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.819080    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.819271 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.819165    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.819271 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.819232    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.872244 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.872209    2554 generic.go:356] "Generic (PLEG): container finished" podID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" containerID="84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729" exitCode=0
Aug 04 18:17:37.872390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.872264    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"
Aug 04 18:17:37.872390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.872306    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerDied","Data":"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"}
Aug 04 18:17:37.872390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.872374    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7" event={"ID":"fc594e62-ff31-4c0c-8df0-b12f4366dedd","Type":"ContainerDied","Data":"b607ee80eee8a70e8ae435d82960fa9ad907a6b1b1836ddc7b40a2c759eb7a62"}
Aug 04 18:17:37.872500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.872410    2554 scope.go:117] "RemoveContainer" containerID="eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"
Aug 04 18:17:37.880716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.880699    2554 scope.go:117] "RemoveContainer" containerID="84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"
Aug 04 18:17:37.888150 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.888129    2554 scope.go:117] "RemoveContainer" containerID="93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"
Aug 04 18:17:37.894718 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.894697    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:17:37.896270 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.896255    2554 scope.go:117] "RemoveContainer" containerID="eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"
Aug 04 18:17:37.896509 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:37.896485    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032\": container with ID starting with eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032 not found: ID does not exist" containerID="eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"
Aug 04 18:17:37.896607 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.896519    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032"} err="failed to get container status \"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032\": rpc error: code = NotFound desc = could not find container \"eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032\": container with ID starting with eef87d7df80166e4f296c010f9aa3acb4ba07f415d6e73a15a00a7c22ad43032 not found: ID does not exist"
Aug 04 18:17:37.896607 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.896543    2554 scope.go:117] "RemoveContainer" containerID="84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"
Aug 04 18:17:37.896838 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:37.896810    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729\": container with ID starting with 84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729 not found: ID does not exist" containerID="84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"
Aug 04 18:17:37.896878 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.896850    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729"} err="failed to get container status \"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729\": rpc error: code = NotFound desc = could not find container \"84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729\": container with ID starting with 84e17abff4806a024ab233a5ae7c5f55153a0df9bb4ed9ed688c5133ad630729 not found: ID does not exist"
Aug 04 18:17:37.896915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.896875    2554 scope.go:117] "RemoveContainer" containerID="93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"
Aug 04 18:17:37.897120 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.897102    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-beeb9e-predictor-6f5c477646-qk7f7"]
Aug 04 18:17:37.897193 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:37.897144    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915\": container with ID starting with 93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915 not found: ID does not exist" containerID="93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"
Aug 04 18:17:37.897193 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.897172    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915"} err="failed to get container status \"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915\": rpc error: code = NotFound desc = could not find container \"93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915\": container with ID starting with 93ccd8ce2adebb671525d5f777beccc59ba36632a0b644ffd8798adb00c5c915 not found: ID does not exist"
Aug 04 18:17:37.920606 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.920576    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.920718 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.920634    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.920718 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.920687    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.920718 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.920711    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljzk\" (UniqueName: \"kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.920873 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:37.920728    2554 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-serving-cert: secret "isvc-init-fail-7f063c-predictor-serving-cert" not found
Aug 04 18:17:37.920873 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.920741    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.920873 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:37.920801    2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls podName:c5fe3a0e-4462-44fb-bb1e-0f8896815513 nodeName:}" failed. No retries permitted until 2026-08-04 18:17:38.420779229 +0000 UTC m=+1035.807469939 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls") pod "isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513") : secret "isvc-init-fail-7f063c-predictor-serving-cert" not found
Aug 04 18:17:37.921087 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.921066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.921352 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.921331    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.921411 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.921356    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.930371 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.930347    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljzk\" (UniqueName: \"kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:37.980144 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:37.980076    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:38.424565 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.424530    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:38.427001 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.426980    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") pod \"isvc-init-fail-7f063c-predictor-65d44b997-vgpm8\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") " pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:38.655983 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.655940    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:38.789396 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.789364    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:38.792092 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:17:38.792063    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5fe3a0e_4462_44fb_bb1e_0f8896815513.slice/crio-c430e0e3845f8eab031960fca99f76aa5b8514721b71a0c752f6b0130322d834 WatchSource:0}: Error finding container c430e0e3845f8eab031960fca99f76aa5b8514721b71a0c752f6b0130322d834: Status 404 returned error can't find the container with id c430e0e3845f8eab031960fca99f76aa5b8514721b71a0c752f6b0130322d834
Aug 04 18:17:38.878999 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.878961    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerStarted","Data":"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"}
Aug 04 18:17:38.879188 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:38.879020    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerStarted","Data":"c430e0e3845f8eab031960fca99f76aa5b8514721b71a0c752f6b0130322d834"}
Aug 04 18:17:39.135539 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:39.135499    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc594e62-ff31-4c0c-8df0-b12f4366dedd" path="/var/lib/kubelet/pods/fc594e62-ff31-4c0c-8df0-b12f4366dedd/volumes"
Aug 04 18:17:40.887348 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:40.887321    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/0.log"
Aug 04 18:17:40.887713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:40.887358    2554 generic.go:356] "Generic (PLEG): container finished" podID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerID="002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599" exitCode=1
Aug 04 18:17:40.887713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:40.887438    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerDied","Data":"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"}
Aug 04 18:17:41.892772 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:41.892734    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/0.log"
Aug 04 18:17:41.893259 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:41.892846    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerStarted","Data":"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"}
Aug 04 18:17:42.647189 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:42.647160    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:42.653045 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:42.652999    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:42.897547 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:42.897451    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer" containerID="cri-o://b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33" gracePeriod=30
Aug 04 18:17:44.338172 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.338151    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/1.log"
Aug 04 18:17:44.338518 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.338505    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/0.log"
Aug 04 18:17:44.340506 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.340491    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:44.369284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369218    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location\") pod \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") "
Aug 04 18:17:44.369284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369250    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert\") pod \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") "
Aug 04 18:17:44.369284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369273    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fljzk\" (UniqueName: \"kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk\") pod \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") "
Aug 04 18:17:44.369476 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369288    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") pod \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") "
Aug 04 18:17:44.369476 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369338    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config\") pod \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\" (UID: \"c5fe3a0e-4462-44fb-bb1e-0f8896815513\") "
Aug 04 18:17:44.369570 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369544    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c5fe3a0e-4462-44fb-bb1e-0f8896815513" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:17:44.369675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369653    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c5fe3a0e-4462-44fb-bb1e-0f8896815513" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:44.369795 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.369753    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c5fe3a0e-4462-44fb-bb1e-0f8896815513" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:44.371459 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.371430    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c5fe3a0e-4462-44fb-bb1e-0f8896815513" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:44.371544 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.371526    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk" (OuterVolumeSpecName: "kube-api-access-fljzk") pod "c5fe3a0e-4462-44fb-bb1e-0f8896815513" (UID: "c5fe3a0e-4462-44fb-bb1e-0f8896815513"). InnerVolumeSpecName "kube-api-access-fljzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:44.470199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.470159    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljzk\" (UniqueName: \"kubernetes.io/projected/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-api-access-fljzk\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:44.470199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.470193    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c5fe3a0e-4462-44fb-bb1e-0f8896815513-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:44.470199 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.470205    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:44.470431 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.470213    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c5fe3a0e-4462-44fb-bb1e-0f8896815513-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:44.470431 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.470222    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c5fe3a0e-4462-44fb-bb1e-0f8896815513-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:44.905593 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.905567    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/1.log"
Aug 04 18:17:44.905925 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.905911    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-7f063c-predictor-65d44b997-vgpm8_c5fe3a0e-4462-44fb-bb1e-0f8896815513/storage-initializer/0.log"
Aug 04 18:17:44.905989 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.905944    2554 generic.go:356] "Generic (PLEG): container finished" podID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerID="b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33" exitCode=1
Aug 04 18:17:44.906048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.906001    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"
Aug 04 18:17:44.906086 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.906050    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerDied","Data":"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"}
Aug 04 18:17:44.906119 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.906101    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8" event={"ID":"c5fe3a0e-4462-44fb-bb1e-0f8896815513","Type":"ContainerDied","Data":"c430e0e3845f8eab031960fca99f76aa5b8514721b71a0c752f6b0130322d834"}
Aug 04 18:17:44.906159 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.906127    2554 scope.go:117] "RemoveContainer" containerID="b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"
Aug 04 18:17:44.915197 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.915181    2554 scope.go:117] "RemoveContainer" containerID="002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"
Aug 04 18:17:44.922772 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.922756    2554 scope.go:117] "RemoveContainer" containerID="b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"
Aug 04 18:17:44.923080 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:44.923060    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33\": container with ID starting with b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33 not found: ID does not exist" containerID="b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"
Aug 04 18:17:44.923134 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.923088    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33"} err="failed to get container status \"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33\": rpc error: code = NotFound desc = could not find container \"b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33\": container with ID starting with b8b6d423c5b638ff81d1c24731733497175d47d09867436074a92cf90ecebc33 not found: ID does not exist"
Aug 04 18:17:44.923134 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.923103    2554 scope.go:117] "RemoveContainer" containerID="002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"
Aug 04 18:17:44.923327 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:17:44.923308    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599\": container with ID starting with 002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599 not found: ID does not exist" containerID="002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"
Aug 04 18:17:44.923369 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.923333    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599"} err="failed to get container status \"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599\": rpc error: code = NotFound desc = could not find container \"002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599\": container with ID starting with 002a9f572aa2050fd1897911e88597c58e53f9e31b832a4ed505f9ac5709b599 not found: ID does not exist"
Aug 04 18:17:44.942987 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.942958    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:44.948723 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:44.948692    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-7f063c-predictor-65d44b997-vgpm8"]
Aug 04 18:17:45.136481 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:45.136449    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" path="/var/lib/kubelet/pods/c5fe3a0e-4462-44fb-bb1e-0f8896815513/volumes"
Aug 04 18:17:47.839391 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.839349    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:47.875558 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875531    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:17:47.875814 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875801    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.875859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875815    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.875859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875832    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.875859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875837    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.875947 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875885    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.875984 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.875974    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5fe3a0e-4462-44fb-bb1e-0f8896815513" containerName="storage-initializer"
Aug 04 18:17:47.878893 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.878874    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.881243 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.881216    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:17:47.881344 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.881292    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fbffe-kube-rbac-proxy-sar-config\""
Aug 04 18:17:47.881576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.881554    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:17:47.881709 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.881686    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-fbffe-predictor-serving-cert\""
Aug 04 18:17:47.891655 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.891636    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:17:47.894477 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.894456    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdshg\" (UniqueName: \"kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.894570 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.894497    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.894629 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.894591    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.894704 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.894680    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.894758 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.894722    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.979313 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.979283    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:47.995284 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995255    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.995425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995297    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.995425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995315    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.995425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995331    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdshg\" (UniqueName: \"kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.995577 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995457    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.995709 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995690    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.996001 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.995982    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.996082 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.996066    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:47.997991 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:47.997972    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:48.004852 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:48.004826    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdshg\" (UniqueName: \"kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg\") pod \"raw-sklearn-fbffe-predictor-586554d447-nnlhq\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") " pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:48.188773 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:48.188687    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:48.321391 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:48.321367    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:17:48.324118 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:17:48.324092    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714fce2a_f603_45e2_8aab_7da4fbf96bad.slice/crio-00c5ced3f7c29f613fdd67673632974fdf8c683cb0ba86c8944c3ca8dbfe93de WatchSource:0}: Error finding container 00c5ced3f7c29f613fdd67673632974fdf8c683cb0ba86c8944c3ca8dbfe93de: Status 404 returned error can't find the container with id 00c5ced3f7c29f613fdd67673632974fdf8c683cb0ba86c8944c3ca8dbfe93de
Aug 04 18:17:48.924548 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:48.924513    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerStarted","Data":"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"}
Aug 04 18:17:48.925016 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:48.924579    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerStarted","Data":"00c5ced3f7c29f613fdd67673632974fdf8c683cb0ba86c8944c3ca8dbfe93de"}
Aug 04 18:17:51.539163 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.539133    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-fe589-predictor-6fbb78db75-45mt9_9ea408dc-c522-4d80-897d-b5e63cf6689b/kserve-container/0.log"
Aug 04 18:17:51.771395 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.771306    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:51.786625 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.786594    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:17:51.786921 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.786892    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kserve-container" containerID="cri-o://f8976ac7383d424b3e1cf4a4fd118cf18c93c1a24ca7aab84a7d16c45464dc9c" gracePeriod=30
Aug 04 18:17:51.787015 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.786898    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kube-rbac-proxy" containerID="cri-o://37d7f923cee3da5c94e2c5ae70beda604c4f339f81a34a31b794f93e2c736b92" gracePeriod=30
Aug 04 18:17:51.845198 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.845165    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:17:51.939811 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.939765    2554 generic.go:356] "Generic (PLEG): container finished" podID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerID="37d7f923cee3da5c94e2c5ae70beda604c4f339f81a34a31b794f93e2c736b92" exitCode=2
Aug 04 18:17:51.939811 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.939805    2554 generic.go:356] "Generic (PLEG): container finished" podID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerID="f8976ac7383d424b3e1cf4a4fd118cf18c93c1a24ca7aab84a7d16c45464dc9c" exitCode=2
Aug 04 18:17:51.940018 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.939833    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerDied","Data":"37d7f923cee3da5c94e2c5ae70beda604c4f339f81a34a31b794f93e2c736b92"}
Aug 04 18:17:51.940018 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:51.939901    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerDied","Data":"f8976ac7383d424b3e1cf4a4fd118cf18c93c1a24ca7aab84a7d16c45464dc9c"}
Aug 04 18:17:52.037574 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.037553    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:17:52.128330 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.128297    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") pod \"9ea408dc-c522-4d80-897d-b5e63cf6689b\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") "
Aug 04 18:17:52.128491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.128356    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s9pwf\" (UniqueName: \"kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf\") pod \"9ea408dc-c522-4d80-897d-b5e63cf6689b\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") "
Aug 04 18:17:52.128491 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.128391    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config\") pod \"9ea408dc-c522-4d80-897d-b5e63cf6689b\" (UID: \"9ea408dc-c522-4d80-897d-b5e63cf6689b\") "
Aug 04 18:17:52.128783 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.128754    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9ea408dc-c522-4d80-897d-b5e63cf6689b" (UID: "9ea408dc-c522-4d80-897d-b5e63cf6689b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:17:52.130483 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.130457    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9ea408dc-c522-4d80-897d-b5e63cf6689b" (UID: "9ea408dc-c522-4d80-897d-b5e63cf6689b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:17:52.130581 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.130559    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf" (OuterVolumeSpecName: "kube-api-access-s9pwf") pod "9ea408dc-c522-4d80-897d-b5e63cf6689b" (UID: "9ea408dc-c522-4d80-897d-b5e63cf6689b"). InnerVolumeSpecName "kube-api-access-s9pwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:17:52.229521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.229494    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9pwf\" (UniqueName: \"kubernetes.io/projected/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-api-access-s9pwf\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:52.229521 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.229520    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9ea408dc-c522-4d80-897d-b5e63cf6689b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:52.229689 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.229530    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9ea408dc-c522-4d80-897d-b5e63cf6689b-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:17:52.945083 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.945022    2554 generic.go:356] "Generic (PLEG): container finished" podID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerID="32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773" exitCode=0
Aug 04 18:17:52.945083 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.945065    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerDied","Data":"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"}
Aug 04 18:17:52.946792 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.946776    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"
Aug 04 18:17:52.946879 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.946779    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9" event={"ID":"9ea408dc-c522-4d80-897d-b5e63cf6689b","Type":"ContainerDied","Data":"62cb9a480969945958d997049baf0c7f41cf057e2d9f1ee25ebd156bbed02f8e"}
Aug 04 18:17:52.946933 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.946897    2554 scope.go:117] "RemoveContainer" containerID="37d7f923cee3da5c94e2c5ae70beda604c4f339f81a34a31b794f93e2c736b92"
Aug 04 18:17:52.956619 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.956597    2554 scope.go:117] "RemoveContainer" containerID="f8976ac7383d424b3e1cf4a4fd118cf18c93c1a24ca7aab84a7d16c45464dc9c"
Aug 04 18:17:52.979344 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.979323    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:17:52.983429 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:52.983394    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-fe589-predictor-6fbb78db75-45mt9"]
Aug 04 18:17:53.137968 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:53.137938    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" path="/var/lib/kubelet/pods/9ea408dc-c522-4d80-897d-b5e63cf6689b/volumes"
Aug 04 18:17:53.952882 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:53.952854    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerStarted","Data":"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"}
Aug 04 18:17:53.953330 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:53.952918    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerStarted","Data":"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"}
Aug 04 18:17:53.953330 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:53.953130    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:53.974688 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:53.974645    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podStartSLOduration=3.213443473 podStartE2EDuration="6.974631348s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.761187875s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:17:53.973829365 +0000 UTC m=+1051.360520097" watchObservedRunningTime="2026-08-04 18:17:53.974631348 +0000 UTC m=+1051.361322079"
Aug 04 18:17:54.958095 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:54.958063    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:17:54.959437 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:54.959408    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:17:55.961594 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:17:55.961549    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:00.966346 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:00.966318    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:18:00.966860 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:00.966832    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:09.343395 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:09.343354    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:10.967779 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:10.967740    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:20.966921 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:20.966840    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:25.688996 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:25.688944    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:25.790130 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:25.790095    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:18:30.967657 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:30.967618    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:40.966831 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:40.966787    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:18:50.966935 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:18:50.966891    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:19:00.967178 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:00.967141    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:19:10.968269 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:10.968239    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:19:18.105777 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.105748    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:19:18.106291 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.105973    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" containerID="cri-o://786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510" gracePeriod=30
Aug 04 18:19:18.106291 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.106063    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kube-rbac-proxy" containerID="cri-o://4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df" gracePeriod=30
Aug 04 18:19:18.131500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.131466    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:18.270745 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.270713    2554 generic.go:356] "Generic (PLEG): container finished" podID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerID="4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df" exitCode=2
Aug 04 18:19:18.270870 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:18.270787    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerDied","Data":"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"}
Aug 04 18:19:20.961991 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:20.961948    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Aug 04 18:19:20.967237 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:20.967211    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Aug 04 18:19:22.442743 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.442718    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:19:22.515649 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515624    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location\") pod \"714fce2a-f603-45e2-8aab-7da4fbf96bad\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") "
Aug 04 18:19:22.515780 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515660    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert\") pod \"714fce2a-f603-45e2-8aab-7da4fbf96bad\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") "
Aug 04 18:19:22.515780 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515692    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config\") pod \"714fce2a-f603-45e2-8aab-7da4fbf96bad\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") "
Aug 04 18:19:22.515780 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515744    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls\") pod \"714fce2a-f603-45e2-8aab-7da4fbf96bad\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") "
Aug 04 18:19:22.515780 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515772    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdshg\" (UniqueName: \"kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg\") pod \"714fce2a-f603-45e2-8aab-7da4fbf96bad\" (UID: \"714fce2a-f603-45e2-8aab-7da4fbf96bad\") "
Aug 04 18:19:22.516017 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.515989    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "714fce2a-f603-45e2-8aab-7da4fbf96bad" (UID: "714fce2a-f603-45e2-8aab-7da4fbf96bad"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:19:22.516017 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.516004    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "714fce2a-f603-45e2-8aab-7da4fbf96bad" (UID: "714fce2a-f603-45e2-8aab-7da4fbf96bad"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:22.516159 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.516092    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "714fce2a-f603-45e2-8aab-7da4fbf96bad" (UID: "714fce2a-f603-45e2-8aab-7da4fbf96bad"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:19:22.517992 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.517959    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg" (OuterVolumeSpecName: "kube-api-access-hdshg") pod "714fce2a-f603-45e2-8aab-7da4fbf96bad" (UID: "714fce2a-f603-45e2-8aab-7da4fbf96bad"). InnerVolumeSpecName "kube-api-access-hdshg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:19:22.518124 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.517984    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "714fce2a-f603-45e2-8aab-7da4fbf96bad" (UID: "714fce2a-f603-45e2-8aab-7da4fbf96bad"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:19:22.616885 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.616851    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/714fce2a-f603-45e2-8aab-7da4fbf96bad-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:19:22.616885 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.616885    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdshg\" (UniqueName: \"kubernetes.io/projected/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-api-access-hdshg\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:19:22.616885 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.616894    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/714fce2a-f603-45e2-8aab-7da4fbf96bad-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:19:22.617081 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.616904    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:19:22.617081 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:22.616912    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/714fce2a-f603-45e2-8aab-7da4fbf96bad-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:19:23.216329 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.216296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:23.274985 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.274953    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:19:23.275226 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275214    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kube-rbac-proxy"
Aug 04 18:19:23.275226 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275227    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kube-rbac-proxy"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275236    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="storage-initializer"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275242    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="storage-initializer"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275250    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kserve-container"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275257    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kserve-container"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275267    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kube-rbac-proxy"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275273    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kube-rbac-proxy"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275279    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275284    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container"
Aug 04 18:19:23.275316 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275322    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kube-rbac-proxy"
Aug 04 18:19:23.275582 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275328    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kserve-container"
Aug 04 18:19:23.275582 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275334    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ea408dc-c522-4d80-897d-b5e63cf6689b" containerName="kserve-container"
Aug 04 18:19:23.275582 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.275342    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerName="kube-rbac-proxy"
Aug 04 18:19:23.279142 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.279126    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.281383 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.281364    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-93f71-predictor-serving-cert\""
Aug 04 18:19:23.281383 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.281377    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-93f71-kube-rbac-proxy-sar-config\""
Aug 04 18:19:23.289376 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.289335    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:19:23.291611 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.291590    2554 generic.go:356] "Generic (PLEG): container finished" podID="714fce2a-f603-45e2-8aab-7da4fbf96bad" containerID="786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510" exitCode=0
Aug 04 18:19:23.291703 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.291620    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerDied","Data":"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"}
Aug 04 18:19:23.291703 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.291672    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq" event={"ID":"714fce2a-f603-45e2-8aab-7da4fbf96bad","Type":"ContainerDied","Data":"00c5ced3f7c29f613fdd67673632974fdf8c683cb0ba86c8944c3ca8dbfe93de"}
Aug 04 18:19:23.291703 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.291697    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"
Aug 04 18:19:23.291828 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.291698    2554 scope.go:117] "RemoveContainer" containerID="4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"
Aug 04 18:19:23.302241 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.302166    2554 scope.go:117] "RemoveContainer" containerID="786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"
Aug 04 18:19:23.310061 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.310016    2554 scope.go:117] "RemoveContainer" containerID="32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"
Aug 04 18:19:23.316758 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.316743    2554 scope.go:117] "RemoveContainer" containerID="4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"
Aug 04 18:19:23.317000 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:19:23.316980    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df\": container with ID starting with 4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df not found: ID does not exist" containerID="4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"
Aug 04 18:19:23.317065 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.317011    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df"} err="failed to get container status \"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df\": rpc error: code = NotFound desc = could not find container \"4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df\": container with ID starting with 4ed282db9de86e12de9f0d16360a78713c3dd0d549a3b31d1d933eb9233aa4df not found: ID does not exist"
Aug 04 18:19:23.317065 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.317045    2554 scope.go:117] "RemoveContainer" containerID="786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"
Aug 04 18:19:23.317311 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:19:23.317291    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510\": container with ID starting with 786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510 not found: ID does not exist" containerID="786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"
Aug 04 18:19:23.317390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.317317    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510"} err="failed to get container status \"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510\": rpc error: code = NotFound desc = could not find container \"786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510\": container with ID starting with 786363082a14bf050eb2bc0bccab8b0bf56df422c5864a2ced2aec68de717510 not found: ID does not exist"
Aug 04 18:19:23.317390 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.317333    2554 scope.go:117] "RemoveContainer" containerID="32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"
Aug 04 18:19:23.317539 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:19:23.317522    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773\": container with ID starting with 32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773 not found: ID does not exist" containerID="32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"
Aug 04 18:19:23.317579 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.317545    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773"} err="failed to get container status \"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773\": rpc error: code = NotFound desc = could not find container \"32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773\": container with ID starting with 32f94ffb76a036c0853a6e9355395c1005badc3ef6f4ffc44d2428902301d773 not found: ID does not exist"
Aug 04 18:19:23.322554 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.322529    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:19:23.325072 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.325053    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-fbffe-predictor-586554d447-nnlhq"]
Aug 04 18:19:23.328950 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.328925    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:19:23.423508 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.423424    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.423508 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.423472    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.423508 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.423500    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.423710 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.423558    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7t4hk\" (UniqueName: \"kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.423710 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.423587    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.524735 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.524690    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.524754    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.524787    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.524814    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7t4hk\" (UniqueName: \"kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.524837    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525319 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.525298    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.525521    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.525586 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.525568    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.527240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.527217    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.535334 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.535315    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7t4hk\" (UniqueName: \"kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk\") pod \"raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.589860 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.589831    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:23.725798 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:23.725639    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:19:23.728409 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:19:23.728384    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edecdcb_bd8d_43b2_95fd_87d037058f3f.slice/crio-cb71b7b55b147201993c91ef1308c947f48859ca48806383dc9c756336eef6cd WatchSource:0}: Error finding container cb71b7b55b147201993c91ef1308c947f48859ca48806383dc9c756336eef6cd: Status 404 returned error can't find the container with id cb71b7b55b147201993c91ef1308c947f48859ca48806383dc9c756336eef6cd
Aug 04 18:19:24.298180 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:24.298148    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerStarted","Data":"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"}
Aug 04 18:19:24.298341 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:24.298206    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerStarted","Data":"cb71b7b55b147201993c91ef1308c947f48859ca48806383dc9c756336eef6cd"}
Aug 04 18:19:25.135426 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:25.135390    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="714fce2a-f603-45e2-8aab-7da4fbf96bad" path="/var/lib/kubelet/pods/714fce2a-f603-45e2-8aab-7da4fbf96bad/volumes"
Aug 04 18:19:28.316639 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:28.316608    2554 generic.go:356] "Generic (PLEG): container finished" podID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerID="6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6" exitCode=0
Aug 04 18:19:28.317011 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:28.316648    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerDied","Data":"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"}
Aug 04 18:19:29.322318 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.322286    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerStarted","Data":"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"}
Aug 04 18:19:29.322694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.322346    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerStarted","Data":"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"}
Aug 04 18:19:29.322694 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.322625    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:29.322778 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.322762    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:29.324016 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.323993    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:19:29.344511 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:29.344467    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podStartSLOduration=2.802020312 podStartE2EDuration="6.344454609s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.542434297s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:19:29.34270228 +0000 UTC m=+1146.729393012" watchObservedRunningTime="2026-08-04 18:19:29.344454609 +0000 UTC m=+1146.731145341"
Aug 04 18:19:30.326283 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:30.326240    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:19:35.330408 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:35.330381    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:19:35.330882 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:35.330827    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:19:45.330915 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:45.330806    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:19:55.333948 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:55.333898    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:19:57.507434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:19:57.507385    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:05.331323 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:05.331285    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:20:15.331528 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:15.331489    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:20:25.330980 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:25.330937    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:20:32.491354 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:32.491306    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:35.331421 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:35.331382    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Aug 04 18:20:37.128332 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:37.128296    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.208294 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:37.208254    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.569038 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:37.568993    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:37.684139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:37.684107    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:39.138959 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:39.138930    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:20:42.611928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:42.611890    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:43.360399 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.360366    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:20:43.360623 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.360593    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container" containerID="cri-o://8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781" gracePeriod=30
Aug 04 18:20:43.360773 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.360645    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kube-rbac-proxy" containerID="cri-o://acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa" gracePeriod=30
Aug 04 18:20:43.385427 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.385398    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:20:43.586248 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.586207    2554 generic.go:356] "Generic (PLEG): container finished" podID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerID="acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa" exitCode=2
Aug 04 18:20:43.586248 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:43.586248    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerDied","Data":"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"}
Aug 04 18:20:45.326568 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:45.326520    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Aug 04 18:20:48.195794 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.195766    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:20:48.320795 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.320759    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config\") pod \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") "
Aug 04 18:20:48.320988 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.320835    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert\") pod \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") "
Aug 04 18:20:48.320988 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.320865    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7t4hk\" (UniqueName: \"kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk\") pod \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") "
Aug 04 18:20:48.320988 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.320892    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls\") pod \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") "
Aug 04 18:20:48.320988 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.320944    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location\") pod \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\" (UID: \"0edecdcb-bd8d-43b2-95fd-87d037058f3f\") "
Aug 04 18:20:48.321344 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.321315    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0edecdcb-bd8d-43b2-95fd-87d037058f3f" (UID: "0edecdcb-bd8d-43b2-95fd-87d037058f3f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:48.321406 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.321317    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0edecdcb-bd8d-43b2-95fd-87d037058f3f" (UID: "0edecdcb-bd8d-43b2-95fd-87d037058f3f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:20:48.321406 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.321365    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0edecdcb-bd8d-43b2-95fd-87d037058f3f" (UID: "0edecdcb-bd8d-43b2-95fd-87d037058f3f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:20:48.323241 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.323217    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0edecdcb-bd8d-43b2-95fd-87d037058f3f" (UID: "0edecdcb-bd8d-43b2-95fd-87d037058f3f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:20:48.323339 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.323257    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk" (OuterVolumeSpecName: "kube-api-access-7t4hk") pod "0edecdcb-bd8d-43b2-95fd-87d037058f3f" (UID: "0edecdcb-bd8d-43b2-95fd-87d037058f3f"). InnerVolumeSpecName "kube-api-access-7t4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:20:48.421498 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.421468    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:20:48.421498 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.421494    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7t4hk\" (UniqueName: \"kubernetes.io/projected/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-api-access-7t4hk\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:20:48.421498 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.421503    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0edecdcb-bd8d-43b2-95fd-87d037058f3f-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:20:48.421717 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.421517    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:20:48.421717 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.421525    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0edecdcb-bd8d-43b2-95fd-87d037058f3f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:20:48.612825 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.612720    2554 generic.go:356] "Generic (PLEG): container finished" podID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerID="8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781" exitCode=0
Aug 04 18:20:48.612825 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.612791    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"
Aug 04 18:20:48.612825 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.612806    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerDied","Data":"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"}
Aug 04 18:20:48.613115 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.612889    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk" event={"ID":"0edecdcb-bd8d-43b2-95fd-87d037058f3f","Type":"ContainerDied","Data":"cb71b7b55b147201993c91ef1308c947f48859ca48806383dc9c756336eef6cd"}
Aug 04 18:20:48.613115 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.612917    2554 scope.go:117] "RemoveContainer" containerID="acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"
Aug 04 18:20:48.621807 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.621790    2554 scope.go:117] "RemoveContainer" containerID="8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"
Aug 04 18:20:48.629561 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.629543    2554 scope.go:117] "RemoveContainer" containerID="6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"
Aug 04 18:20:48.636746 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.636728    2554 scope.go:117] "RemoveContainer" containerID="acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"
Aug 04 18:20:48.637084 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:20:48.637021    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa\": container with ID starting with acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa not found: ID does not exist" containerID="acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"
Aug 04 18:20:48.637139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637098    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa"} err="failed to get container status \"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa\": rpc error: code = NotFound desc = could not find container \"acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa\": container with ID starting with acbb27c524765d29029caa0e75e5a823fd9b738778e569ed8d41f14f105bf4fa not found: ID does not exist"
Aug 04 18:20:48.637139 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637123    2554 scope.go:117] "RemoveContainer" containerID="8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"
Aug 04 18:20:48.637367 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:20:48.637348    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781\": container with ID starting with 8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781 not found: ID does not exist" containerID="8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"
Aug 04 18:20:48.637404 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637375    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781"} err="failed to get container status \"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781\": rpc error: code = NotFound desc = could not find container \"8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781\": container with ID starting with 8a95134f5476f91b0f1af6777c0edc6d295128d88826ef59b5adae0a3f2da781 not found: ID does not exist"
Aug 04 18:20:48.637404 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637390    2554 scope.go:117] "RemoveContainer" containerID="6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"
Aug 04 18:20:48.637639 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:20:48.637621    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6\": container with ID starting with 6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6 not found: ID does not exist" containerID="6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"
Aug 04 18:20:48.637684 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637643    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6"} err="failed to get container status \"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6\": rpc error: code = NotFound desc = could not find container \"6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6\": container with ID starting with 6efe429e8546bb6fad5ec4462c0854dc9f27d999d0411c0e4bf519c0b27775a6 not found: ID does not exist"
Aug 04 18:20:48.637849 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.637831    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:20:48.642082 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:48.642062    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-93f71-predictor-8455bcbcdc-rvlwk"]
Aug 04 18:20:49.135346 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:20:49.135318    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" path="/var/lib/kubelet/pods/0edecdcb-bd8d-43b2-95fd-87d037058f3f/volumes"
Aug 04 18:21:57.264322 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:21:57.264286    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:21:57.326536 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:21:57.326500    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:02.445527 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.445487    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:02.512130 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512095    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:22:02.512400 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512387    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.512448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512402    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.512448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512417    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container"
Aug 04 18:22:02.512448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512422    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container"
Aug 04 18:22:02.512448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512429    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="storage-initializer"
Aug 04 18:22:02.512448 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512435    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="storage-initializer"
Aug 04 18:22:02.512595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512484    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kube-rbac-proxy"
Aug 04 18:22:02.512595 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.512492    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0edecdcb-bd8d-43b2-95fd-87d037058f3f" containerName="kserve-container"
Aug 04 18:22:02.515586 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.515568    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.518017 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.517991    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Aug 04 18:22:02.518017 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.517998    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:22:02.518210 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.518020    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Aug 04 18:22:02.518360 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.518326    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Aug 04 18:22:02.518488 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.518475    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:22:02.519087 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.519070    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:22:02.519177 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.519099    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Aug 04 18:22:02.524293 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.524272    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:22:02.533394 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.533369    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:02.605058 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.605007    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:02.649901 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.649861    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.650092 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.649909    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.650092 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.649940    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.650092 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.649967    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64vgt\" (UniqueName: \"kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.650092 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.649995    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.707292 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.707177    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:22:02.751309 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751279    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.751434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751319    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.751434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751346    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.751434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751377    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64vgt\" (UniqueName: \"kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.751434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751405    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.751848 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.751824    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.752066 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.752018    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.752138 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.752080    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.753935 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.753917    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.761044 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.761007    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64vgt\" (UniqueName: \"kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt\") pod \"isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.826526 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.826501    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:02.961838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:02.961808    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:22:02.963986 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:22:02.963958    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cac7b9b_ae32_4e5c_aa07_9173eb76bc1a.slice/crio-0fb1ef091d958f42ba3395f689868dd6925c5fbebbc5f74fd497bff17ccbe85b WatchSource:0}: Error finding container 0fb1ef091d958f42ba3395f689868dd6925c5fbebbc5f74fd497bff17ccbe85b: Status 404 returned error can't find the container with id 0fb1ef091d958f42ba3395f689868dd6925c5fbebbc5f74fd497bff17ccbe85b
Aug 04 18:22:03.885425 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:03.885391    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerStarted","Data":"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"}
Aug 04 18:22:03.885796 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:03.885455    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerStarted","Data":"0fb1ef091d958f42ba3395f689868dd6925c5fbebbc5f74fd497bff17ccbe85b"}
Aug 04 18:22:06.896091 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:06.896056    2554 generic.go:356] "Generic (PLEG): container finished" podID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerID="edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224" exitCode=0
Aug 04 18:22:06.896091 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:06.896093    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerDied","Data":"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"}
Aug 04 18:22:07.902048 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.901995    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerStarted","Data":"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"}
Aug 04 18:22:07.902525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.902089    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerStarted","Data":"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"}
Aug 04 18:22:07.902525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.902377    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:07.902525 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.902429    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:07.903834 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.903808    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:07.930060 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:07.929999    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podStartSLOduration=2.210041639 podStartE2EDuration="5.929984834s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.719943195s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:22:07.928265105 +0000 UTC m=+1305.314955837" watchObservedRunningTime="2026-08-04 18:22:07.929984834 +0000 UTC m=+1305.316675566"
Aug 04 18:22:08.905574 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:08.905536    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:13.910169 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:13.910137    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:22:13.910660 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:13.910632    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:23.910551 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:23.910513    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:33.910910 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:33.910824    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:43.910845 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:43.910764    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:22:53.911211 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:22:53.911165    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:23:03.911391 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:23:03.911347    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:23:13.910622 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:23:13.910585    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Aug 04 18:23:23.911190 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:23:23.911154    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:23:27.514382 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:23:27.514327    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:27.271700 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:27.271610    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:27.377682 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:27.377649    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:47.793839 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:47.793806    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:24:47.794323 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:47.794204    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container" containerID="cri-o://479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2" gracePeriod=30
Aug 04 18:24:47.794323 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:47.794241    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kube-rbac-proxy" containerID="cri-o://d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c" gracePeriod=30
Aug 04 18:24:47.821902 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:47.821859    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:24:48.465533 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:48.465498    2554 generic.go:356] "Generic (PLEG): container finished" podID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerID="d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c" exitCode=2
Aug 04 18:24:48.465699 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:48.465564    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerDied","Data":"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"}
Aug 04 18:24:48.906120 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:48.906085    2554 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Aug 04 18:24:52.137250 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.137225    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:24:52.242501 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242417    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64vgt\" (UniqueName: \"kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt\") pod \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") "
Aug 04 18:24:52.242501 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242466    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config\") pod \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") "
Aug 04 18:24:52.242853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242510    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert\") pod \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") "
Aug 04 18:24:52.242853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242559    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls\") pod \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") "
Aug 04 18:24:52.242853 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242587    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location\") pod \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\" (UID: \"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a\") "
Aug 04 18:24:52.243022 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242913    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" (UID: "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:52.243022 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242909    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" (UID: "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:24:52.243022 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.242907    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" (UID: "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:24:52.244799 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.244775    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt" (OuterVolumeSpecName: "kube-api-access-64vgt") pod "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" (UID: "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a"). InnerVolumeSpecName "kube-api-access-64vgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:24:52.244876 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.244800    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" (UID: "0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:24:52.343507 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.343475    2554 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-cabundle-cert\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:24:52.343507 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.343504    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:24:52.343696 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.343516    2554 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kserve-provision-location\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:24:52.343696 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.343530    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64vgt\" (UniqueName: \"kubernetes.io/projected/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-api-access-64vgt\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:24:52.343696 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.343542    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:24:52.480507 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.480474    2554 generic.go:356] "Generic (PLEG): container finished" podID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerID="479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2" exitCode=0
Aug 04 18:24:52.480674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.480542    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerDied","Data":"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"}
Aug 04 18:24:52.480674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.480590    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg" event={"ID":"0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a","Type":"ContainerDied","Data":"0fb1ef091d958f42ba3395f689868dd6925c5fbebbc5f74fd497bff17ccbe85b"}
Aug 04 18:24:52.480674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.480600    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"
Aug 04 18:24:52.480674 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.480617    2554 scope.go:117] "RemoveContainer" containerID="d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"
Aug 04 18:24:52.489073 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.489047    2554 scope.go:117] "RemoveContainer" containerID="479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"
Aug 04 18:24:52.496079 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.496062    2554 scope.go:117] "RemoveContainer" containerID="edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"
Aug 04 18:24:52.502972 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.502956    2554 scope.go:117] "RemoveContainer" containerID="d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"
Aug 04 18:24:52.503285 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:24:52.503257    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c\": container with ID starting with d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c not found: ID does not exist" containerID="d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"
Aug 04 18:24:52.503350 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.503287    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c"} err="failed to get container status \"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c\": rpc error: code = NotFound desc = could not find container \"d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c\": container with ID starting with d6c730ddd36fd4c6c3066c44dcd7555c5c058944345fa8c17264da5e8d4f581c not found: ID does not exist"
Aug 04 18:24:52.503350 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.503311    2554 scope.go:117] "RemoveContainer" containerID="479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"
Aug 04 18:24:52.503573 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:24:52.503556    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2\": container with ID starting with 479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2 not found: ID does not exist" containerID="479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"
Aug 04 18:24:52.503656 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.503583    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2"} err="failed to get container status \"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2\": rpc error: code = NotFound desc = could not find container \"479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2\": container with ID starting with 479bc47b66a556bd01b830627128d58810e9f15fc9bc72485afd3ea682dbeba2 not found: ID does not exist"
Aug 04 18:24:52.503656 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.503606    2554 scope.go:117] "RemoveContainer" containerID="edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"
Aug 04 18:24:52.503836 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:24:52.503819    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224\": container with ID starting with edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224 not found: ID does not exist" containerID="edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"
Aug 04 18:24:52.503891 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.503843    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224"} err="failed to get container status \"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224\": rpc error: code = NotFound desc = could not find container \"edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224\": container with ID starting with edda3bf87af692a592bc01a8b2db958593e46c01ada2b86cc72fbbbec8526224 not found: ID does not exist"
Aug 04 18:24:52.504429 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.504412    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:24:52.510548 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:52.510528    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-8bcd7648f-9b5dg"]
Aug 04 18:24:53.136378 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:24:53.136346    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" path="/var/lib/kubelet/pods/0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a/volumes"
Aug 04 18:25:01.518364 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.518325    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:01.554490 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554458    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:25:01.554721 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554709    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="storage-initializer"
Aug 04 18:25:01.554768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554723    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="storage-initializer"
Aug 04 18:25:01.554768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554733    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container"
Aug 04 18:25:01.554768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554738    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container"
Aug 04 18:25:01.554768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554754    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kube-rbac-proxy"
Aug 04 18:25:01.554768 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554759    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kube-rbac-proxy"
Aug 04 18:25:01.554941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554798    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kserve-container"
Aug 04 18:25:01.554941 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.554806    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cac7b9b-ae32-4e5c-aa07-9173eb76bc1a" containerName="kube-rbac-proxy"
Aug 04 18:25:01.559470 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.559452    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.561835 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.561814    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-kube-rbac-proxy-sar-config\""
Aug 04 18:25:01.561948 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.561856    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-cipn-predictor-serving-cert\""
Aug 04 18:25:01.561948 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.561856    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Aug 04 18:25:01.562058 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.561998    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-vsp4f\""
Aug 04 18:25:01.562843 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.562829    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Aug 04 18:25:01.567838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.567818    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:25:01.608356 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.608327    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.608484 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.608363    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.608484 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.608383    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7fvl\" (UniqueName: \"kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.690137 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.690100    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:01.709208 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.709182    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.709336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.709216    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.709336 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.709236    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7fvl\" (UniqueName: \"kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.709807 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.709785    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.711701 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.711683    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.718562 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.718540    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7fvl\" (UniqueName: \"kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl\") pod \"message-dumper-raw-cipn-predictor-9774bf55d-j55j2\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") " pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.869626 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.869599    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:01.996638 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:01.996612    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:25:01.998923 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:25:01.998902    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf64c02c6_1e08_4bcb_8def_046b07c92862.slice/crio-d188a8f99195c24a8cf462c139bc6989c95a6f1c34320339c281e353badda827 WatchSource:0}: Error finding container d188a8f99195c24a8cf462c139bc6989c95a6f1c34320339c281e353badda827: Status 404 returned error can't find the container with id d188a8f99195c24a8cf462c139bc6989c95a6f1c34320339c281e353badda827
Aug 04 18:25:02.519881 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:02.519847    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerStarted","Data":"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"}
Aug 04 18:25:02.520301 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:02.519910    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerStarted","Data":"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"}
Aug 04 18:25:02.520301 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:02.519964    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerStarted","Data":"d188a8f99195c24a8cf462c139bc6989c95a6f1c34320339c281e353badda827"}
Aug 04 18:25:02.520301 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:02.519991    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:02.538291 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:02.538250    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" podStartSLOduration=1.538234404 podStartE2EDuration="1.538234404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:25:02.537077851 +0000 UTC m=+1479.923768584" watchObservedRunningTime="2026-08-04 18:25:02.538234404 +0000 UTC m=+1479.924925175"
Aug 04 18:25:03.523504 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:03.523470    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:03.524895 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:03.524876    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:10.534127 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:10.534096    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:25:11.551394 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:11.551345    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:25:11.989117 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:25:11.989065    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:36.594158 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:36.594123    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-cipn-predictor-9774bf55d-j55j2_f64c02c6-1e08-4bcb-8def-046b07c92862/kserve-container/0.log"
Aug 04 18:26:36.810353 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:36.810322    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:26:36.810594 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:36.810551    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kserve-container" containerID="cri-o://c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa" gracePeriod=30
Aug 04 18:26:36.810717 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:36.810588    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kube-rbac-proxy" containerID="cri-o://dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a" gracePeriod=30
Aug 04 18:26:37.046893 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.046866    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:26:37.090100 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.090072    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config\") pod \"f64c02c6-1e08-4bcb-8def-046b07c92862\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") "
Aug 04 18:26:37.090219 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.090120    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7fvl\" (UniqueName: \"kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl\") pod \"f64c02c6-1e08-4bcb-8def-046b07c92862\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") "
Aug 04 18:26:37.090219 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.090148    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls\") pod \"f64c02c6-1e08-4bcb-8def-046b07c92862\" (UID: \"f64c02c6-1e08-4bcb-8def-046b07c92862\") "
Aug 04 18:26:37.090418 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.090395    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f64c02c6-1e08-4bcb-8def-046b07c92862" (UID: "f64c02c6-1e08-4bcb-8def-046b07c92862"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Aug 04 18:26:37.092307 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.092279    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f64c02c6-1e08-4bcb-8def-046b07c92862" (UID: "f64c02c6-1e08-4bcb-8def-046b07c92862"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Aug 04 18:26:37.092421 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.092322    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl" (OuterVolumeSpecName: "kube-api-access-q7fvl") pod "f64c02c6-1e08-4bcb-8def-046b07c92862" (UID: "f64c02c6-1e08-4bcb-8def-046b07c92862"). InnerVolumeSpecName "kube-api-access-q7fvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:26:37.191264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.191208    2554 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-rbac-proxy-sar-config\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:26:37.191264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.191230    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7fvl\" (UniqueName: \"kubernetes.io/projected/f64c02c6-1e08-4bcb-8def-046b07c92862-kube-api-access-q7fvl\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:26:37.191264 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.191239    2554 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f64c02c6-1e08-4bcb-8def-046b07c92862-proxy-tls\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:26:37.595263 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595231    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wvz5s/must-gather-5x5bt"]
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595470    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kserve-container"
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595481    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kserve-container"
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595502    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kube-rbac-proxy"
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595509    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kube-rbac-proxy"
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595548    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kserve-container"
Aug 04 18:26:37.595615 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.595556    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerName="kube-rbac-proxy"
Aug 04 18:26:37.598434 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.598418    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.600003 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.599978    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:37.600687 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.600669    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wvz5s\"/\"default-dockercfg-b59jb\""
Aug 04 18:26:37.600776 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.600703    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvz5s\"/\"kube-root-ca.crt\""
Aug 04 18:26:37.600838 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.600780    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wvz5s\"/\"openshift-service-ca.crt\""
Aug 04 18:26:37.619366 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.619345    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvz5s/must-gather-5x5bt"]
Aug 04 18:26:37.669174 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.669145    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:26:37.694774 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.694749    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtjt\" (UniqueName: \"kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.694872 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.694793    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.795490 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.795463    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.795576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.795504    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtjt\" (UniqueName: \"kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.795791 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.795774    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.815811 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.815789    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtjt\" (UniqueName: \"kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt\") pod \"must-gather-5x5bt\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") " pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:37.861567 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861503    2554 generic.go:356] "Generic (PLEG): container finished" podID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerID="dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a" exitCode=2
Aug 04 18:26:37.861567 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861527    2554 generic.go:356] "Generic (PLEG): container finished" podID="f64c02c6-1e08-4bcb-8def-046b07c92862" containerID="c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa" exitCode=2
Aug 04 18:26:37.861567 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861555    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"
Aug 04 18:26:37.861713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861582    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerDied","Data":"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"}
Aug 04 18:26:37.861713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861643    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerDied","Data":"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"}
Aug 04 18:26:37.861713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861665    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2" event={"ID":"f64c02c6-1e08-4bcb-8def-046b07c92862","Type":"ContainerDied","Data":"d188a8f99195c24a8cf462c139bc6989c95a6f1c34320339c281e353badda827"}
Aug 04 18:26:37.861713 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.861691    2554 scope.go:117] "RemoveContainer" containerID="dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"
Aug 04 18:26:37.874417 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.874398    2554 scope.go:117] "RemoveContainer" containerID="c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"
Aug 04 18:26:37.881379 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.881349    2554 scope.go:117] "RemoveContainer" containerID="dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"
Aug 04 18:26:37.881634 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:26:37.881614    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a\": container with ID starting with dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a not found: ID does not exist" containerID="dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"
Aug 04 18:26:37.881681 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.881641    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"} err="failed to get container status \"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a\": rpc error: code = NotFound desc = could not find container \"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a\": container with ID starting with dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a not found: ID does not exist"
Aug 04 18:26:37.881681 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.881657    2554 scope.go:117] "RemoveContainer" containerID="c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"
Aug 04 18:26:37.881907 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:26:37.881890    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa\": container with ID starting with c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa not found: ID does not exist" containerID="c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"
Aug 04 18:26:37.881956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.881912    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"} err="failed to get container status \"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa\": rpc error: code = NotFound desc = could not find container \"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa\": container with ID starting with c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa not found: ID does not exist"
Aug 04 18:26:37.881956 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.881927    2554 scope.go:117] "RemoveContainer" containerID="dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"
Aug 04 18:26:37.882153 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.882135    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a"} err="failed to get container status \"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a\": rpc error: code = NotFound desc = could not find container \"dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a\": container with ID starting with dd3df36a569033739d564c877e5bac494b16e867e3313c9704df144039ed231a not found: ID does not exist"
Aug 04 18:26:37.882193 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.882154    2554 scope.go:117] "RemoveContainer" containerID="c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"
Aug 04 18:26:37.882371 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.882353    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa"} err="failed to get container status \"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa\": rpc error: code = NotFound desc = could not find container \"c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa\": container with ID starting with c064a1f49161c9b0c0df97d0af5e40ba33cba32bcbd32041ffaeb51f8f6019fa not found: ID does not exist"
Aug 04 18:26:37.885592 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.885574    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:26:37.895616 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.895597    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-cipn-predictor-9774bf55d-j55j2"]
Aug 04 18:26:37.907639 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:37.907620    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:26:38.042538 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:38.042507    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wvz5s/must-gather-5x5bt"]
Aug 04 18:26:38.045638 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:26:38.045614    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce1589b5_3c5b_4d97_8355_2953a890ab84.slice/crio-65de8e74e33d8e0ede84c0baaa680dbd1fec2aa4543e7c01dc9c501b7d6902a8 WatchSource:0}: Error finding container 65de8e74e33d8e0ede84c0baaa680dbd1fec2aa4543e7c01dc9c501b7d6902a8: Status 404 returned error can't find the container with id 65de8e74e33d8e0ede84c0baaa680dbd1fec2aa4543e7c01dc9c501b7d6902a8
Aug 04 18:26:38.047784 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:38.047757    2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Aug 04 18:26:38.867716 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:38.867663    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" event={"ID":"ce1589b5-3c5b-4d97-8355-2953a890ab84","Type":"ContainerStarted","Data":"65de8e74e33d8e0ede84c0baaa680dbd1fec2aa4543e7c01dc9c501b7d6902a8"}
Aug 04 18:26:39.139610 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:39.139184    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f64c02c6-1e08-4bcb-8def-046b07c92862" path="/var/lib/kubelet/pods/f64c02c6-1e08-4bcb-8def-046b07c92862/volumes"
Aug 04 18:26:44.893524 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:44.893478    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" event={"ID":"ce1589b5-3c5b-4d97-8355-2953a890ab84","Type":"ContainerStarted","Data":"4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e"}
Aug 04 18:26:44.893892 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:44.893537    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" event={"ID":"ce1589b5-3c5b-4d97-8355-2953a890ab84","Type":"ContainerStarted","Data":"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"}
Aug 04 18:26:44.909932 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:26:44.909880    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" podStartSLOduration=1.698204286 podStartE2EDuration="7.909865942s" totalImagesPullingTime="6.211661656s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-08-04 18:26:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:26:44.909746199 +0000 UTC m=+1582.296436933" watchObservedRunningTime="2026-08-04 18:26:44.909865942 +0000 UTC m=+1582.296556674"
Aug 04 18:27:06.981605 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:06.981523    2554 generic.go:356] "Generic (PLEG): container finished" podID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerID="031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78" exitCode=0
Aug 04 18:27:06.982046 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:06.981597    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" event={"ID":"ce1589b5-3c5b-4d97-8355-2953a890ab84","Type":"ContainerDied","Data":"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"}
Aug 04 18:27:06.982046 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:06.981910    2554 scope.go:117] "RemoveContainer" containerID="031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"
Aug 04 18:27:07.582182 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:07.582148    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wvz5s_must-gather-5x5bt_ce1589b5-3c5b-4d97-8355-2953a890ab84/gather/0.log"
Aug 04 18:27:08.045085 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:08.044979    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:08.239994 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:08.239957    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:08.305130 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:08.305023    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:10.943308 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:10.943276    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hv2mv_290108f0-7082-4602-827e-23aa0cc5015c/global-pull-secret-syncer/0.log"
Aug 04 18:27:11.051862 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:11.051825    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log"
Aug 04 18:27:11.182751 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:11.182720    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-15.ec2.internal_bf92005c4ad1b4b9a80ced15063e26f5/haproxy/0.log"
Aug 04 18:27:13.011041 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.010997    2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-wvz5s/must-gather-5x5bt"]
Aug 04 18:27:13.011457 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.011191    2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-wvz5s/must-gather-5x5bt" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="copy" containerID="cri-o://4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e" gracePeriod=2
Aug 04 18:27:13.013700 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:27:13.013659    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5x5bt\" is forbidden: User \"system:node:ip-10-0-140-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wvz5s\": no relationship found between node 'ip-10-0-140-15.ec2.internal' and this object" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:27:13.014102 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.014078    2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-wvz5s/must-gather-5x5bt"]
Aug 04 18:27:13.135794 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:27:13.135752    2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-5x5bt\" is forbidden: User \"system:node:ip-10-0-140-15.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-wvz5s\": no relationship found between node 'ip-10-0-140-15.ec2.internal' and this object" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:27:13.244365 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.244342    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wvz5s_must-gather-5x5bt_ce1589b5-3c5b-4d97-8355-2953a890ab84/copy/0.log"
Aug 04 18:27:13.245851 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.245834    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:27:13.384522 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.384495    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output\") pod \"ce1589b5-3c5b-4d97-8355-2953a890ab84\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") "
Aug 04 18:27:13.384667 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.384567    2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjtjt\" (UniqueName: \"kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt\") pod \"ce1589b5-3c5b-4d97-8355-2953a890ab84\" (UID: \"ce1589b5-3c5b-4d97-8355-2953a890ab84\") "
Aug 04 18:27:13.386479 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.386457    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "ce1589b5-3c5b-4d97-8355-2953a890ab84" (UID: "ce1589b5-3c5b-4d97-8355-2953a890ab84"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Aug 04 18:27:13.386800 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.386774    2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt" (OuterVolumeSpecName: "kube-api-access-fjtjt") pod "ce1589b5-3c5b-4d97-8355-2953a890ab84" (UID: "ce1589b5-3c5b-4d97-8355-2953a890ab84"). InnerVolumeSpecName "kube-api-access-fjtjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Aug 04 18:27:13.485949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.485919    2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjtjt\" (UniqueName: \"kubernetes.io/projected/ce1589b5-3c5b-4d97-8355-2953a890ab84-kube-api-access-fjtjt\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:27:13.485949 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.485945    2554 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/ce1589b5-3c5b-4d97-8355-2953a890ab84-must-gather-output\") on node \"ip-10-0-140-15.ec2.internal\" DevicePath \"\""
Aug 04 18:27:13.905214 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:13.905179    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:14.015770 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.015733    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-wvz5s_must-gather-5x5bt_ce1589b5-3c5b-4d97-8355-2953a890ab84/copy/0.log"
Aug 04 18:27:14.016240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.016109    2554 generic.go:356] "Generic (PLEG): container finished" podID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerID="4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e" exitCode=143
Aug 04 18:27:14.016240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.016215    2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wvz5s/must-gather-5x5bt"
Aug 04 18:27:14.016240 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.016229    2554 scope.go:117] "RemoveContainer" containerID="4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e"
Aug 04 18:27:14.024185 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.024166    2554 scope.go:117] "RemoveContainer" containerID="031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"
Aug 04 18:27:14.032502 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.032477    2554 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5n7p9_46df8a4b-d4c2-4a7a-ac67-705e1445e84b/konnectivity-agent/0.log" limit=1024
Aug 04 18:27:14.035217 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.035197    2554 scope.go:117] "RemoveContainer" containerID="4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e"
Aug 04 18:27:14.035489 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:27:14.035468    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e\": container with ID starting with 4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e not found: ID does not exist" containerID="4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e"
Aug 04 18:27:14.035546 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.035496    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e"} err="failed to get container status \"4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e\": rpc error: code = NotFound desc = could not find container \"4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e\": container with ID starting with 4a17ec6c38bdd45dfa70d76b55629646e075f80e9a16d2a98a92b23224e4820e not found: ID does not exist"
Aug 04 18:27:14.035546 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.035514    2554 scope.go:117] "RemoveContainer" containerID="031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"
Aug 04 18:27:14.035772 ip-10-0-140-15 kubenswrapper[2554]: E0804 18:27:14.035747    2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78\": container with ID starting with 031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78 not found: ID does not exist" containerID="031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"
Aug 04 18:27:14.035834 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:14.035781    2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78"} err="failed to get container status \"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78\": rpc error: code = NotFound desc = could not find container \"031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78\": container with ID starting with 031c0c9b25b6fe77b3e3536812931b2b78302cae8ee4f31670603a4f4bd40e78 not found: ID does not exist"
Aug 04 18:27:15.135040 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:15.135004    2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" path="/var/lib/kubelet/pods/ce1589b5-3c5b-4d97-8355-2953a890ab84/volumes"
Aug 04 18:27:15.634558 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:15.634524    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2qg7_3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547/node-exporter/0.log"
Aug 04 18:27:15.665576 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:15.665551    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2qg7_3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547/kube-rbac-proxy/0.log"
Aug 04 18:27:15.690765 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:15.690735    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-g2qg7_3ec04b12-0baf-4bb1-bd7f-5f6ab15c9547/init-textfile/0.log"
Aug 04 18:27:18.623319 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623290    2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"]
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623533    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="copy"
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623543    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="copy"
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623560    2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="gather"
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623565    2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="gather"
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623621    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="gather"
Aug 04 18:27:18.623675 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.623629    2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1589b5-3c5b-4d97-8355-2953a890ab84" containerName="copy"
Aug 04 18:27:18.627643 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.627623    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.629963 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.629931    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flh2v\"/\"kube-root-ca.crt\""
Aug 04 18:27:18.629963 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.629955    2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-flh2v\"/\"default-dockercfg-g5r45\""
Aug 04 18:27:18.630180 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.629932    2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-flh2v\"/\"openshift-service-ca.crt\""
Aug 04 18:27:18.636121 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.636100    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"]
Aug 04 18:27:18.726575 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.726543    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-podres\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.726733 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.726589    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-sys\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.726733 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.726616    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg7j7\" (UniqueName: \"kubernetes.io/projected/ae0c545e-768a-45af-b605-4547127d4dbb-kube-api-access-bg7j7\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.726733 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.726635    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-lib-modules\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.726733 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.726650    2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-proc\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827754 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827723    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-sys\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827775    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg7j7\" (UniqueName: \"kubernetes.io/projected/ae0c545e-768a-45af-b605-4547127d4dbb-kube-api-access-bg7j7\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827803    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-lib-modules\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827827    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-proc\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827844    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-sys\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827871    2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-podres\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.827928 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827905    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-proc\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.828211 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.827974    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-lib-modules\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.828211 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.828012    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ae0c545e-768a-45af-b605-4547127d4dbb-podres\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.837238 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.837219    2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg7j7\" (UniqueName: \"kubernetes.io/projected/ae0c545e-768a-45af-b605-4547127d4dbb-kube-api-access-bg7j7\") pod \"perf-node-gather-daemonset-lvjr9\" (UID: \"ae0c545e-768a-45af-b605-4547127d4dbb\") " pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:18.937338 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:18.937261    2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:19.062914 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:19.062890    2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"]
Aug 04 18:27:19.065650 ip-10-0-140-15 kubenswrapper[2554]: W0804 18:27:19.065625    2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podae0c545e_768a_45af_b605_4547127d4dbb.slice/crio-24458ed7efd8e4bf5646e5ab4e2589a3c37912db8e8faadf489337b4d714d8e6 WatchSource:0}: Error finding container 24458ed7efd8e4bf5646e5ab4e2589a3c37912db8e8faadf489337b4d714d8e6: Status 404 returned error can't find the container with id 24458ed7efd8e4bf5646e5ab4e2589a3c37912db8e8faadf489337b4d714d8e6
Aug 04 18:27:19.294997 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:19.294964    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5b8776669-9sbvq_35dc7b5d-4102-49e7-9fbf-3dd75627a2eb/volume-data-source-validator/0.log"
Aug 04 18:27:20.045946 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.045906    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9" event={"ID":"ae0c545e-768a-45af-b605-4547127d4dbb","Type":"ContainerStarted","Data":"f44d9c80d3a9f99b9cb5b5221c8ba2ec3934b475dbce40dbf18f24b3c9b9f8d7"}
Aug 04 18:27:20.046537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.045996    2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9" event={"ID":"ae0c545e-768a-45af-b605-4547127d4dbb","Type":"ContainerStarted","Data":"24458ed7efd8e4bf5646e5ab4e2589a3c37912db8e8faadf489337b4d714d8e6"}
Aug 04 18:27:20.046537 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.046079    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:20.066041 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.065980    2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9" podStartSLOduration=2.065965037 podStartE2EDuration="2.065965037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-08-04 18:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-08-04 18:27:20.064601547 +0000 UTC m=+1617.451292281" watchObservedRunningTime="2026-08-04 18:27:20.065965037 +0000 UTC m=+1617.452655771"
Aug 04 18:27:20.235808 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.235775    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbzvp_5658ae88-67e4-4545-ae55-5a352c56be47/dns/0.log"
Aug 04 18:27:20.258766 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.258745    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbzvp_5658ae88-67e4-4545-ae55-5a352c56be47/kube-rbac-proxy/0.log"
Aug 04 18:27:20.330709 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.330681    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gfq2x_ca064d63-4126-457e-b8ba-240ef6524fc6/dns-node-resolver/0.log"
Aug 04 18:27:20.851804 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:20.851773    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-z9znh_3cee3f9c-fdf7-4d1c-ac51-bf894c2b36ea/node-ca/0.log"
Aug 04 18:27:21.627721 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:21.627694    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-64dcfc95cb-lqxzk_9429ca73-a4a0-41a3-a154-5426e63032c9/router/0.log"
Aug 04 18:27:22.038294 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:22.038214    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mz9vp_74d02d17-8838-4cfc-b54c-4c0e85fd2093/serve-healthcheck-canary/0.log"
Aug 04 18:27:22.436900 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:22.436877    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwkr7_52fba404-4a23-4553-803a-0a1c47fc074d/kube-rbac-proxy/0.log"
Aug 04 18:27:22.462944 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:22.462921    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwkr7_52fba404-4a23-4553-803a-0a1c47fc074d/exporter/0.log"
Aug 04 18:27:22.486215 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:22.486181    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gwkr7_52fba404-4a23-4553-803a-0a1c47fc074d/extractor/0.log"
Aug 04 18:27:24.806282 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:24.806252    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-5f9d7f69b6-ssbn8_e4276eda-9d2c-472f-b753-b0b7ccfc1ced/manager/0.log"
Aug 04 18:27:26.059178 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:26.059152    2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-flh2v/perf-node-gather-daemonset-lvjr9"
Aug 04 18:27:29.309410 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:29.309376    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-t2dfj_7bf910e7-72eb-4fb2-a499-7c69ee708b20/migrator/0.log"
Aug 04 18:27:29.349565 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:29.349534    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-t2dfj_7bf910e7-72eb-4fb2-a499-7c69ee708b20/graceful-termination/0.log"
Aug 04 18:27:29.719437 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:29.719322    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-x789v_754f774e-53e8-4b98-bea2-8c64eeea8f28/kube-storage-version-migrator-operator/1.log"
Aug 04 18:27:29.720859 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:29.720833    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7878f4b88c-x789v_754f774e-53e8-4b98-bea2-8c64eeea8f28/kube-storage-version-migrator-operator/0.log"
Aug 04 18:27:30.824436 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.824406    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/kube-multus-additional-cni-plugins/0.log"
Aug 04 18:27:30.845274 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.845243    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/egress-router-binary-copy/0.log"
Aug 04 18:27:30.866399 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.866374    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/cni-plugins/0.log"
Aug 04 18:27:30.887500 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.887474    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/bond-cni-plugin/0.log"
Aug 04 18:27:30.907672 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.907637    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/routeoverride-cni/0.log"
Aug 04 18:27:30.928692 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.928664    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/whereabouts-cni-bincopy/0.log"
Aug 04 18:27:30.952096 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:30.952071    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xq9l_0995f211-3886-4776-9e68-8c40f2a1cc0e/whereabouts-cni/0.log"
Aug 04 18:27:31.152699 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:31.152623    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-gwx7r_03456812-0405-4710-965d-a544cc7be2ef/kube-multus/0.log"
Aug 04 18:27:31.249609 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:31.249577    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hsmdl_4bea9f6f-a036-4ccb-bc75-5214012b9a70/network-metrics-daemon/0.log"
Aug 04 18:27:31.270376 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:31.270345    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-hsmdl_4bea9f6f-a036-4ccb-bc75-5214012b9a70/kube-rbac-proxy/0.log"
Aug 04 18:27:32.294374 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.294346    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/ovn-controller/0.log"
Aug 04 18:27:32.311820 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.311797    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/ovn-acl-logging/0.log"
Aug 04 18:27:32.341224 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.341199    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/kube-rbac-proxy-node/0.log"
Aug 04 18:27:32.364668 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.364643    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/kube-rbac-proxy-ovn-metrics/0.log"
Aug 04 18:27:32.384503 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.384480    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/northd/0.log"
Aug 04 18:27:32.407292 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.407273    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/nbdb/0.log"
Aug 04 18:27:32.428979 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.428961    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/sbdb/0.log"
Aug 04 18:27:32.482467 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:32.482447    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-njg2j_0100536f-3d05-49fd-973c-1d04c047f8d6/ovnkube-controller/0.log"
Aug 04 18:27:33.775412 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:33.775383    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wc9ks_69a6ac8e-d331-4b39-9046-19ad6bead317/network-check-target-container/0.log"
Aug 04 18:27:34.667011 ip-10-0-140-15 kubenswrapper[2554]: I0804 18:27:34.666987    2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-fqvgx_3902a019-e7cf-4612-b8f5-edb9db0d881c/iptables-alerter/0.log"