Jul 28 17:14:56.475531 ip-10-0-134-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:14:56.475543 ip-10-0-134-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:14:56.475550 ip-10-0-134-28 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:14:56.475800 ip-10-0-134-28 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:15:06.618303 ip-10-0-134-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:15:06.618323 ip-10-0-134-28 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 98caac065fd14cc9835ebcdbada28ebe --
Jul 28 17:16:38.291251 ip-10-0-134-28 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:16:38.733629 ip-10-0-134-28 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:38.733629 ip-10-0-134-28 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:16:38.733629 ip-10-0-134-28 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:38.733629 ip-10-0-134-28 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:16:38.733629 ip-10-0-134-28 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:16:38.734334 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.733702    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:16:38.738619 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738603    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738620    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738625    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738629    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738632    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738635    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738638    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738641    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738644    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738647    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738651    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738655    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738658    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738660    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738663    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738666    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.738662 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738669    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738673    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738675    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738678    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738680    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738683    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738686    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738689    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738697    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738701    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738705    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738708    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738711    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738714    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738717    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738719    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738722    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738725    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738727    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.739045 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738730    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738733    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738735    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738738    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738740    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738743    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738745    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738748    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738750    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738753    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738755    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738758    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738760    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738762    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738765    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738768    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738771    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738774    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738777    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738779    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.739536 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738782    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738785    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738787    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738790    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738793    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738795    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738798    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738801    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738804    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738806    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738809    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738811    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738813    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738816    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738818    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738821    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738824    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738827    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738829    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738832    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.740031 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738835    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738837    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738839    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738842    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738844    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738847    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738849    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738853    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738856    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738860    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738863    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738865    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738868    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738871    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738873    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738876    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738878    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738881    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738884    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738886    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.740500 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738889    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738892    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738894    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.738897    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740435    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740442    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740446    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740449    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740452    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740458    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740461    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740463    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740466    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740468    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740471    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740473    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740476    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740479    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740481    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.740975 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740485    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740488    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740490    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740493    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740496    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740498    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740501    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740504    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740507    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740509    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740512    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740515    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740517    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740520    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740523    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740525    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740528    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740530    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740533    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740535    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.741434 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740538    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740540    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740543    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740546    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740549    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740551    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740554    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740556    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740559    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740561    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740565    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740569    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740573    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740576    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740579    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740581    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740584    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740586    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740589    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.741928 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740592    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740595    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740598    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740600    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740603    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740606    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740610    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740613    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740616    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740619    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740621    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740625    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740627    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740630    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740633    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740636    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740639    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740643    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740645    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740648    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.742392 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740651    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740653    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740656    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740658    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740661    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740666    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740668    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740671    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740673    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740676    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740679    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740681    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740684    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740686    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740689    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740691    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740694    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740697    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740699    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.742898 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740702    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740706    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740709    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740712    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740715    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.740718    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741102    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741108    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741112    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741115    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741119    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741122    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741125    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741127    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741130    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741133    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741136    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741138    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741142    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741144    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.743397 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741147    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741151    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741154    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741157    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741160    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741163    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741165    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741168    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741171    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741174    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741176    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741179    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741181    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741184    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741186    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741189    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741192    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741195    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741197    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.743881 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741200    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741204    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741208    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741210    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741213    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741216    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741219    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741221    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741223    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741226    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741229    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741231    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741234    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741237    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741239    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741242    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741244    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741247    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741249    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741252    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.744385 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741254    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741257    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741260    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741262    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741264    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741267    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741269    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741272    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741274    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741277    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741280    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741283    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741285    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741288    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741290    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741293    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741295    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741298    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741302    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.744872 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741305    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741308    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741311    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741314    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741317    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741320    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741322    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741325    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741327    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741330    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741332    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741335    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741337    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741340    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741343    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741345    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741348    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741350    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741353    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741355    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.745338 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741358    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741361    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741363    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741366    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741369    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741371    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.741374    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741442    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741449    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741455    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741459    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741464    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741467    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741471    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741475    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741479    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741482    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741486    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741489    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741492    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741495    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741498    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741501    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:16:38.745836 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741504    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741507    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741511    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741514    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741517    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741520    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741524    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741527    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741531    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741536    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741539    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741542    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741545    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741548    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741551    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741554    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741558    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741561    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741564    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741567    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741570    2576 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741574    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741578    2576 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741581    2576 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:16:38.746398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741584    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741589    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741592    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741595    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741598    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741601    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741604    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741607    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741610    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741612    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741616    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741619    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741621    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741624    2576 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741628    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741631    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741634    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741638    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741641    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741644    2576 flags.go:64] FLAG: --help="false"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741647    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-134-28.ec2.internal"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741650    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741653    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:16:38.747008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741655    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741659    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741662    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741665    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741668    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741671    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741674    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741677    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741680    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741683    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741687    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741690    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741693    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741696    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741699    2576 flags.go:64] FLAG: --lock-file=""
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741702    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741704    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741707    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741713    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741715    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741718    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741721    2576 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741723    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741727    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:16:38.747607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741730    2576 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741733    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741738    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741741    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741745    2576 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741748    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741751    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741753    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741756    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741759    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741762    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741769    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741775    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741779    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741782    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741785    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741790    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741795    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741799    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741802    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741806    2576 flags.go:64] FLAG: --port="10250"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741809    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741812    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09df9ff34bdb8e877"
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741815    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:16:38.748190 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741818    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741821    2576 flags.go:64] FLAG: --register-node="true"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741824    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741827    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741830    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741833    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741836    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741839    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741842    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741845    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741848    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741852    2576 flags.go:64] FLAG: --runonce="false"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741855    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741858    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741861    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741864    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741867    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741869    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741873    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741875    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741878    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741881    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741884    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741886    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741889    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:16:38.748770 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741893    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741897    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741902    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741905    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741908    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741912    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741915    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741918    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741921    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741923    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741926    2576 flags.go:64] FLAG: --v="2"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741930    2576 flags.go:64] FLAG: --version="false"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741934    2576 flags.go:64] FLAG: --vmodule=""
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741938    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.741941    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742063    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742067    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742070    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742073    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742076    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742079    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742082    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742085    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.749427 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742087    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742090    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742093    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742095    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742098    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742101    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742103    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742106    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742108    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742111    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742115    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742118    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742121    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742123    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742126    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742128    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742131    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742133    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742136    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742138    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.750017 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742141    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742145    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742148    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742151    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742154    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742156    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742159    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742162    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742165    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742168    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742170    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742173    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742175    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742178    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742180    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742183    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742185    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742188    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742190    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742193    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.750499 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742195    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742198    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742201    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742204    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742207    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742209    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742212    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742214    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742217    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742220    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742222    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742225    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742227    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742230    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742232    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742235    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742237    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742240    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742242    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742245    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.750987 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742247    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742250    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742254    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742257    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742261    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742263    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742267    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742270    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742273    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742276    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742279    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742282    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742284    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742287    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742291    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742293    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742296    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742299    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742301    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.751560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742304    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742306    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742309    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742311    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742314    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742316    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742319    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742321    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742324    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742326    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742329    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.742331    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.743335    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.751279    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.751299    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:16:38.752097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.751313    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.751323    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751379    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751384    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751387    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751391    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751395    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751397    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751400    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751403    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751406    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751408    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751411    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751414    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751417    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751419    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751422    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751424    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751427    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.752464 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751429    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751432    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751436    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751440    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751443    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751446    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751449    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751452    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751455    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751457    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751461    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751465    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751469    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751472    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751475    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751478    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751480    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751483    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751486    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.752919 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751488    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751491    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751493    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751495    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751498    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751501    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751503    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751506    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751508    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751511    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751513    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751516    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751518    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751521    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751523    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751526    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751528    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751531    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751533    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751536    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.753387 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751538    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751541    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751544    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751546    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751549    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751552    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751555    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751557    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751560    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751563    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751567    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751571    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751573    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751576    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751578    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751581    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751583    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751586    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751588    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751591    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.753867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751593    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751596    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751598    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751601    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751604    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751606    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751609    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751611    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751614    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751616    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751618    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751621    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751623    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751626    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751628    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751631    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751634    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751637    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751639    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751643    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.754355 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751645    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751648    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751650    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.751655    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751762    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751767    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751770    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751773    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751776    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751779    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751782    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751784    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751787    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751789    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751792    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:16:38.755048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751794    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751797    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751800    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751802    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751805    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751809    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751812    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751815    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751817    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751821    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751824    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751827    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751830    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751833    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751836    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751839    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751842    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751845    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751848    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:16:38.755709 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751850    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751853    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751855    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751858    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751860    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751863    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751865    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751869    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751872    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751875    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751878    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751881    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751883    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751886    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751889    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751892    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751895    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751897    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751900    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751903    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:16:38.756185 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751905    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751908    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751910    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751913    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751915    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751918    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751920    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751923    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751926    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751928    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751931    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751934    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751936    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751939    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751941    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751944    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751946    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751948    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751951    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751953    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:16:38.756704 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751956    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751958    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751960    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751976    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751979    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751981    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751984    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751986    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751989    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751992    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751995    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751997    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.751999    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752002    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752004    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752007    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752009    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752012    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752014    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:16:38.757208 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752017    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752021    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752023    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752026    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752028    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752031    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752034    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752036    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752039    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:38.752041    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.752046    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.752142    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:16:38.757702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.754413    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:16:38.759207 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.759191    2576 server.go:1019] "Starting client certificate rotation"
Jul 28 17:16:38.759323 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.759306    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:38.759355 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.759340    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:16:38.762055 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.762042    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:38.763153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.763137    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:16:38.772748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.772731    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:16:38.775023 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.775010    2576 log.go:25] "Validated CRI v1 image API"
Jul 28 17:16:38.775813 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.775796    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:16:38.782269 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.782248    2576 fs.go:135] Filesystem UUIDs: map[3bec5c09-f379-4a40-a11a-9aecb843996a:/dev/nvme0n1p4 5cde2de3-d540-4ec4-9d3b-cbf9def81a88:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 28 17:16:38.782364 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.782269    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:16:38.788753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.788611    2576 manager.go:216] Machine: {Timestamp:2026-07-28 17:16:38.787197946 +0000 UTC m=+0.369368827 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:2999382 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec212eeceea250b9444e1c8bca3c1968 SystemUUID:ec212eec-eea2-50b9-444e-1c8bca3c1968 BootID:98caac06-5fd1-4cc9-835e-bcdbada28ebe Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:3c:2f:12:87:a7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:3c:2f:12:87:a7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:31:77:53:f3:3d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:16:38.788753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.788740    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:16:38.788916 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.788843    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:16:38.791072 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.791050    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:38.792291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.792266    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:16:38.792431 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.792294    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-134-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:16:38.793140 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.793129    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:16:38.793140 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.793140    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:16:38.793240 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.793148    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:38.794046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.794036    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:16:38.795232 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.795223    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:16:38.797061 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797050    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:38.797167 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797158    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:16:38.797203 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797187    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:16:38.797203 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797201    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:16:38.797264 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797216    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:16:38.797264 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797226    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:16:38.797264 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797234    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:16:38.797604 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797583    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:38.797648 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.797611    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:16:38.800189 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.800170    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:16:38.802640 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.802616    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:16:38.802724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.802654    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:16:38.804776 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804763    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804785    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804791    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804797    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804805    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804811    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804819    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:16:38.804827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804824    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:16:38.805037 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804834    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:16:38.805037 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804840    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:16:38.805037 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804848    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:16:38.805037 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.804853    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:16:38.805686 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.805676    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:16:38.805686 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.805686    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:16:38.806938 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.806915    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-134-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:16:38.806938 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.806916    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:16:38.808713 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.808701    2576 server.go:1268] "Started kubelet"
Jul 28 17:16:38.808801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.808764    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:16:38.808850 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.808787    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:16:38.808898 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.808853    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:16:38.809510 ip-10-0-134-28 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:16:38.810880 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.810863    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:16:38.811590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.811440    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:16:38.817439 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.817421    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:16:38.820940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.820922    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:16:38.821043 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.820928    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:38.821043 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.821026    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:38.821702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821684    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:16:38.821702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821684    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:16:38.821859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821707    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:16:38.821859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821738    2576 factory.go:55] Registering systemd factory
Jul 28 17:16:38.821859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821751    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:16:38.821859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821803    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:16:38.821859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821829    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:16:38.822096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821941    2576 factory.go:153] Registering CRI-O factory
Jul 28 17:16:38.822096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.821951    2576 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:16:38.822096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.822012    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:16:38.822096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.822031    2576 factory.go:103] Registering Raw factory
Jul 28 17:16:38.822096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.822043    2576 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:16:38.822491 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.822477    2576 manager.go:349] Starting recovery of all containers
Jul 28 17:16:38.823252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.823235    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-134-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:16:38.825430 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.823350    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18c68399c80f880b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-07-28 17:16:38.808676363 +0000 UTC m=+0.390847242,LastTimestamp:2026-07-28 17:16:38.808676363 +0000 UTC m=+0.390847242,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Jul 28 17:16:38.834398 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.834357    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-134-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:16:38.834518 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.834396    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:16:38.836849 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.836829    2576 manager.go:354] Recovery completed
Jul 28 17:16:38.843011 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.842992    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:38.845153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845136    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:38.845215 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845167    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:38.845215 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845177    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:38.845693 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845679    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:16:38.845731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845694    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:16:38.845731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845701    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:38.845731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.845711    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:16:38.846372 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.846359    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:38.846425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.846382    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:38.846425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.846392    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:38.847197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.847135    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18c68399ca3c1d04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-07-28 17:16:38.845152516 +0000 UTC m=+0.427323397,LastTimestamp:2026-07-28 17:16:38.845152516 +0000 UTC m=+0.427323397,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Jul 28 17:16:38.847662 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.847650    2576 policy_none.go:49] "None policy: Start"
Jul 28 17:16:38.847712 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.847666    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:16:38.847712 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.847677    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:16:38.848865 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.848854    2576 policy_none.go:47] "Start"
Jul 28 17:16:38.857796 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.857591    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18c68399ca3c6789  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-134-28.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-07-28 17:16:38.845171593 +0000 UTC m=+0.427342477,LastTimestamp:2026-07-28 17:16:38.845171593 +0000 UTC m=+0.427342477,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Jul 28 17:16:38.861960 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.861940    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ggms5"
Jul 28 17:16:38.868829 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.868747    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18c68399ca3c8af8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-134-28.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-07-28 17:16:38.845180664 +0000 UTC m=+0.427351546,LastTimestamp:2026-07-28 17:16:38.845180664 +0000 UTC m=+0.427351546,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Jul 28 17:16:38.868940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.868927    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ggms5"
Jul 28 17:16:38.871782 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.871697    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-134-28.ec2.internal.18c68399ca3c1d04\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-134-28.ec2.internal.18c68399ca3c1d04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-134-28.ec2.internal,UID:ip-10-0-134-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-134-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-134-28.ec2.internal,},FirstTimestamp:2026-07-28 17:16:38.845152516 +0000 UTC m=+0.427323397,LastTimestamp:2026-07-28 17:16:38.846371131 +0000 UTC m=+0.428542012,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-134-28.ec2.internal,}"
Jul 28 17:16:38.882476 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.882463    2576 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:16:38.882558 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.882489    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:16:38.882558 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.882498    2576 server.go:87] "Starting device plugin registration server"
Jul 28 17:16:38.882734 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.882720    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:16:38.882814 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.882734    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:16:38.882950 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.882935    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:16:38.883053 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.883039    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:16:38.883183 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.883054    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:16:38.883407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.883390    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:16:38.883474 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.883442    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:38.897497 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.897477    2576 container_manager_linux.go:676] "Failed to ensure state" containerName="/system.slice" err="failed to move PID 2610 into the system container \"/system.slice\": <nil>"
Jul 28 17:16:38.955452 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.955426    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:16:38.956670 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.956649    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:16:38.956741 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.956687    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:16:38.956741 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.956711    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:16:38.956833 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.956740    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:16:38.958858 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.958835    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:38.983401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.983381    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:38.984363 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.984319    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:38.984363 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.984352    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:38.984496 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.984368    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:38.984496 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.984396    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-134-28.ec2.internal"
Jul 28 17:16:38.995467 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:38.995452    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-134-28.ec2.internal"
Jul 28 17:16:38.995538 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:38.995471    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-134-28.ec2.internal\": node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:39.056831 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.056797    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"]
Jul 28 17:16:39.056948 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.056872    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:39.058455 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.058438    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:39.058553 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.058468    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:39.058553 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.058481    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:39.059758 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.059744    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:39.059889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.059878    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.059935 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.059902    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:39.060445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060426    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:39.060533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060467    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:39.060533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060486    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:39.060533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060499    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:39.060533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060469    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:39.060672 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.060547    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:39.061665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.061651    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.061720 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.061677    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:39.062362 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.062335    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:39.062437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.062376    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:39.062437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.062390    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:39.064264 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.064249    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:39.091521 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.091502    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-28.ec2.internal\" not found" node="ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.095888 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.095874    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-134-28.ec2.internal\" not found" node="ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.123354 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.123334    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.123448 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.123365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.123448 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.123391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0f6325ce07165ebf2e9979367b54cf4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"f0f6325ce07165ebf2e9979367b54cf4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.164530 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.164496    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:39.223917 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.223889    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.224035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.223858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.224035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.223960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.224035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.224001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f8670d84f658402c20684d057cb8783-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal\" (UID: \"0f8670d84f658402c20684d057cb8783\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.224035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.224007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0f6325ce07165ebf2e9979367b54cf4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"f0f6325ce07165ebf2e9979367b54cf4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.224035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.224035    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/f0f6325ce07165ebf2e9979367b54cf4-config\") pod \"kube-apiserver-proxy-ip-10-0-134-28.ec2.internal\" (UID: \"f0f6325ce07165ebf2e9979367b54cf4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.265188 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.265129    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:39.365830 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.365802    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:39.394006 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.393961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.399816 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:39.399798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:39.466109 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:39.466083    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:40.962213 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:40.962191    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.062635 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.062606    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.154088 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.154059    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:16:41.154637 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.154226    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:16:41.163329 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.163313    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.263559 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.263517    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.270675 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.270643    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:11:40 +0000 UTC" deadline="2028-04-19 05:13:42.387824008 +0000 UTC"
Jul 28 17:16:41.270675 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.270672    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15131h57m1.117154952s"
Jul 28 17:16:41.319117 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.319086    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:16:41.320340 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.320324    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:16:41.320401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.320355    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:16:41.320401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.320369    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:16:41.342896 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:41.342857    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8670d84f658402c20684d057cb8783.slice/crio-4bb3286e2b384a60fd1dc4ebd79422d43c6188c1c95f2257cd57188665702352 WatchSource:0}: Error finding container 4bb3286e2b384a60fd1dc4ebd79422d43c6188c1c95f2257cd57188665702352: Status 404 returned error can't find the container with id 4bb3286e2b384a60fd1dc4ebd79422d43c6188c1c95f2257cd57188665702352
Jul 28 17:16:41.343383 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:41.343365    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f6325ce07165ebf2e9979367b54cf4.slice/crio-ac89f08c9cc0f7b7e0b7b00b2f9f1d2ee73d6059ed0ccf88411c458083d61f1f WatchSource:0}: Error finding container ac89f08c9cc0f7b7e0b7b00b2f9f1d2ee73d6059ed0ccf88411c458083d61f1f: Status 404 returned error can't find the container with id ac89f08c9cc0f7b7e0b7b00b2f9f1d2ee73d6059ed0ccf88411c458083d61f1f
Jul 28 17:16:41.349237 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.349217    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:16:41.355819 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.355779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"0f8670d84f658402c20684d057cb8783","Type":"ContainerStarted","Data":"4bb3286e2b384a60fd1dc4ebd79422d43c6188c1c95f2257cd57188665702352"}
Jul 28 17:16:41.356823 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.356806    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" event={"ID":"f0f6325ce07165ebf2e9979367b54cf4","Type":"ContainerStarted","Data":"ac89f08c9cc0f7b7e0b7b00b2f9f1d2ee73d6059ed0ccf88411c458083d61f1f"}
Jul 28 17:16:41.364192 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.364172    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.464680 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.464648    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.565198 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.565170    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.612286 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.612251    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:41.665731 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.665711    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.679617 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.679588    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:41.766648 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:41.766616    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-134-28.ec2.internal\" not found"
Jul 28 17:16:41.778621 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.778591    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:41.816960 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.816894    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:41.831647 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.831617    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:41.832720 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.832688    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal"
Jul 28 17:16:41.841939 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:41.841923    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:16:42.115844 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.115768    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:16:42.193977 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.193949    2576 apiserver.go:52] "Watching apiserver"
Jul 28 17:16:42.202326 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.202298    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:16:42.202728 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.202698    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5swwd","kube-system/konnectivity-agent-qg46g","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h","openshift-cluster-node-tuning-operator/tuned-jqtw4","openshift-multus/multus-additional-cni-plugins-g2rl4","openshift-multus/network-metrics-daemon-tqq8n","openshift-network-operator/iptables-alerter-zxntr","kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal","openshift-dns/node-resolver-647fs","openshift-image-registry/node-ca-w5blq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal","openshift-multus/multus-q24qv","openshift-network-diagnostics/network-check-target-k5p9w","openshift-ovn-kubernetes/ovnkube-node-vcdd5"]
Jul 28 17:16:42.204185 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.204164    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.205295 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.205268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:42.205388 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.205352    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:42.207026 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.206716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.207026 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.206771    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:16:42.207026 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.206845    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-p5czz\""
Jul 28 17:16:42.207026 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.206773    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.207262 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.207219    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.208992 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.208500    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.209181 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.209162    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.209289 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.209255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.209483 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.209466    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:16:42.209640 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.209538    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6hp24\""
Jul 28 17:16:42.210263 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.210244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.211072 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.211016    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sshjh\""
Jul 28 17:16:42.211158 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.211086    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.211227 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.211203    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.211445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.211422    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.211516 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.211494    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:42.212952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.212736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.212952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.212867    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:16:42.212952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.212895    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.212952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.212909    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:16:42.212952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.212934    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6sq4\""
Jul 28 17:16:42.213217 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.213136    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.213738 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.213724    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:16:42.214033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.214013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.214111 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.214088    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:42.215124 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.215104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.215558 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.215528    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.215721 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.215705    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.215809 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.215760    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hw2rj\""
Jul 28 17:16:42.216449 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.216434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.217562 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.217524    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:16:42.217652 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.217603    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cwk8s\""
Jul 28 17:16:42.217768 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.217752    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:16:42.217939 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.217921    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:16:42.218440 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.218421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.219707 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.219637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.219707 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.219649    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:16:42.219888 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.219777    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.220373    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8769f\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.221023    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.221149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gs794\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.222048    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.222343    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.222669    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:16:42.222968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.222923    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:16:42.224197 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.224178    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:16:42.224852 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.224835    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.227333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.227316    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:16:42.227433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.227411    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rchxq\""
Jul 28 17:16:42.229488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229467    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-device-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.229589 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-os-release\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.229589 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-host\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.229589 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkpkf\" (UniqueName: \"kubernetes.io/projected/4b5333fe-432d-40d7-a893-35a95b16181f-kube-api-access-xkpkf\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.229757 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.229757 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-kubelet-config\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.229757 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229797    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-run\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229827    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-tmp\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229853    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-config\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw9c8\" (UniqueName: \"kubernetes.io/projected/8267b0b1-b622-489a-878d-6061b47da927-kube-api-access-kw9c8\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df4df771-c44e-4a2f-a59a-f80f87f2e19c-agent-certs\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-slash\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.229999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.229985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-systemd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-lib-modules\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-env-overrides\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-sys-fs\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230138    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-sys\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysconfig\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230300    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-systemd\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.230329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-etc-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230344    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hcp\" (UniqueName: \"kubernetes.io/projected/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-kube-api-access-m5hcp\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230365    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-var-lib-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230379    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-script-lib\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230396    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230419    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-dbus\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cnibin\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230535    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8267b0b1-b622-489a-878d-6061b47da927-hosts-file\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55868a3b-2011-4bda-8189-bdfdfebb3273-host-slash\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230620    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6px5\" (UniqueName: \"kubernetes.io/projected/55868a3b-2011-4bda-8189-bdfdfebb3273-kube-api-access-z6px5\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-socket-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230643    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230701    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-kubelet\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230731    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs5c6\" (UniqueName: \"kubernetes.io/projected/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-kube-api-access-fs5c6\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.230857 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-var-lib-kubelet\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230784    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-node-log\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230829    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230904    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-systemd-units\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-ovn\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.230976    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231020    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khmxp\" (UniqueName: \"kubernetes.io/projected/89932144-cfe2-4f4c-a545-64c463f6d879-kube-api-access-khmxp\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231070    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-kubernetes\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231092    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-registration-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-system-cni-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-log-socket\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55868a3b-2011-4bda-8189-bdfdfebb3273-iptables-alerter-script\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231225    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8267b0b1-b622-489a-878d-6061b47da927-tmp-dir\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df4df771-c44e-4a2f-a59a-f80f87f2e19c-konnectivity-ca\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.231660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231279    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-conf\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-host\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231330    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-etc-tuned\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6jjn\" (UniqueName: \"kubernetes.io/projected/43e86123-b87c-4297-acaa-ba1de2d293de-kube-api-access-k6jjn\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-netd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-serviceca\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231420    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-modprobe-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231441    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-netns\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-bin\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.232437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.231485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8j5x\" (UniqueName: \"kubernetes.io/projected/d37aad0c-f620-44a7-9396-a8cdf7379c15-kube-api-access-m8j5x\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.318217 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.318145    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:16:42.332647 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-lib-modules\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332656    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-env-overrides\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-socket-dir-parent\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332707    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-sys-fs\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-k8s-cni-cncf-io\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-etc-kubernetes\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.332798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332801    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332822    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-sys\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332832    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-lib-modules\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332867    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-netns\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-sys-fs\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.332998    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-sys\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333065    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333093    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysconfig\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-systemd\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-etc-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333174    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-cnibin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hcp\" (UniqueName: \"kubernetes.io/projected/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-kube-api-access-m5hcp\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333214    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-var-lib-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333238    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-script-lib\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-env-overrides\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-dbus\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333427    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cnibin\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8267b0b1-b622-489a-878d-6061b47da927-hosts-file\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.333536    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333597    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-var-lib-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.333637    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:16:42.833597916 +0000 UTC m=+3.020191201 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333655    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55868a3b-2011-4bda-8189-bdfdfebb3273-host-slash\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333742    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6px5\" (UniqueName: \"kubernetes.io/projected/55868a3b-2011-4bda-8189-bdfdfebb3273-kube-api-access-z6px5\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-socket-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333553    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-systemd\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.333821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-kubelet\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cnibin\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs5c6\" (UniqueName: \"kubernetes.io/projected/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-kube-api-access-fs5c6\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-var-lib-kubelet\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333923    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysconfig\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/55868a3b-2011-4bda-8189-bdfdfebb3273-host-slash\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-etc-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-cni-binary-copy\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.333982    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334027    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-kubelet\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334042    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8267b0b1-b622-489a-878d-6061b47da927-hosts-file\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-dbus\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-var-lib-kubelet\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-openvswitch\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-node-log\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.334465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334281    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-script-lib\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334284    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-etc-selinux\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-socket-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-node-log\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-cni-binary-copy\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.334993    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-systemd-units\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-ovn\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.335030    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-os-release\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.335071    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:42.835058567 +0000 UTC m=+3.021651855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335082    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-systemd-units\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-ovn\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khmxp\" (UniqueName: \"kubernetes.io/projected/89932144-cfe2-4f4c-a545-64c463f6d879-kube-api-access-khmxp\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335130    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335153    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-kubernetes\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.335266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335211    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-kubernetes\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335307    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-multus\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-ovn-kubernetes\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4s2\" (UniqueName: \"kubernetes.io/projected/02612ec5-b216-495d-92ab-3569ec0ee16e-kube-api-access-8l4s2\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-registration-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335497    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-system-cni-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-log-socket\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-registration-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335576    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-kubelet\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335586    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-system-cni-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55868a3b-2011-4bda-8189-bdfdfebb3273-iptables-alerter-script\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-log-socket\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8267b0b1-b622-489a-878d-6061b47da927-tmp-dir\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df4df771-c44e-4a2f-a59a-f80f87f2e19c-konnectivity-ca\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335683    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-conf\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-host\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-etc-tuned\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6jjn\" (UniqueName: \"kubernetes.io/projected/43e86123-b87c-4297-acaa-ba1de2d293de-kube-api-access-k6jjn\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-netd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335809    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-serviceca\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-modprobe-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-netns\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335902    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-bin\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335942    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8j5x\" (UniqueName: \"kubernetes.io/projected/d37aad0c-f620-44a7-9396-a8cdf7379c15-kube-api-access-m8j5x\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-system-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.335991    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-hostroot\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8267b0b1-b622-489a-878d-6061b47da927-tmp-dir\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-device-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336063    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/89932144-cfe2-4f4c-a545-64c463f6d879-device-dir\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-os-release\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-host\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336150    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkpkf\" (UniqueName: \"kubernetes.io/projected/4b5333fe-432d-40d7-a893-35a95b16181f-kube-api-access-xkpkf\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336178    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-daemon-config\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.336752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336182    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-sysctl-conf\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336212    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/55868a3b-2011-4bda-8189-bdfdfebb3273-iptables-alerter-script\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-host\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-kubelet-config\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336246    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df4df771-c44e-4a2f-a59a-f80f87f2e19c-konnectivity-ca\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336267    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/9a6a32c3-2640-4cf8-8a4f-daf938694431-kubelet-config\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336294    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-run\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-run-netns\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-tmp\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-os-release\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336344    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-config\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-host\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336374    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-bin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336412    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-multus-certs\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336441    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw9c8\" (UniqueName: \"kubernetes.io/projected/8267b0b1-b622-489a-878d-6061b47da927-kube-api-access-kw9c8\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.337513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336465    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df4df771-c44e-4a2f-a59a-f80f87f2e19c-agent-certs\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-slash\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-run\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-netd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/43e86123-b87c-4297-acaa-ba1de2d293de-etc-modprobe-d\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-tuning-conf-dir\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-cni-bin\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.336939    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-host-slash\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.337050    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-serviceca\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.337255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-systemd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.337284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-conf-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.337392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d37aad0c-f620-44a7-9396-a8cdf7379c15-run-systemd\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.337703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovnkube-config\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338311 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.338068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d37aad0c-f620-44a7-9396-a8cdf7379c15-ovn-node-metrics-cert\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.338915 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.338520    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-etc-tuned\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.338915 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.338820    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/43e86123-b87c-4297-acaa-ba1de2d293de-tmp\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.340447 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.340417    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df4df771-c44e-4a2f-a59a-f80f87f2e19c-agent-certs\") pod \"konnectivity-agent-qg46g\" (UID: \"df4df771-c44e-4a2f-a59a-f80f87f2e19c\") " pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.342255 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.342235    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:42.342255 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.342257    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:42.342449 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.342270    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:42.342449 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.342327    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:42.842312469 +0000 UTC m=+3.028905769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:42.343247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.343218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6px5\" (UniqueName: \"kubernetes.io/projected/55868a3b-2011-4bda-8189-bdfdfebb3273-kube-api-access-z6px5\") pod \"iptables-alerter-zxntr\" (UID: \"55868a3b-2011-4bda-8189-bdfdfebb3273\") " pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.343660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.343636    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hcp\" (UniqueName: \"kubernetes.io/projected/07acdd3f-1248-4bb5-ab18-ad1cdf17f694-kube-api-access-m5hcp\") pod \"multus-additional-cni-plugins-g2rl4\" (UID: \"07acdd3f-1248-4bb5-ab18-ad1cdf17f694\") " pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.344518 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.344496    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khmxp\" (UniqueName: \"kubernetes.io/projected/89932144-cfe2-4f4c-a545-64c463f6d879-kube-api-access-khmxp\") pod \"aws-ebs-csi-driver-node-ttg7h\" (UID: \"89932144-cfe2-4f4c-a545-64c463f6d879\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.345177 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.345125    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs5c6\" (UniqueName: \"kubernetes.io/projected/b05ffd10-3b28-4ae6-9fd5-89b15f0f4139-kube-api-access-fs5c6\") pod \"node-ca-w5blq\" (UID: \"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139\") " pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.345791 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.345769    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8j5x\" (UniqueName: \"kubernetes.io/projected/d37aad0c-f620-44a7-9396-a8cdf7379c15-kube-api-access-m8j5x\") pod \"ovnkube-node-vcdd5\" (UID: \"d37aad0c-f620-44a7-9396-a8cdf7379c15\") " pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.346432 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.346413    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw9c8\" (UniqueName: \"kubernetes.io/projected/8267b0b1-b622-489a-878d-6061b47da927-kube-api-access-kw9c8\") pod \"node-resolver-647fs\" (UID: \"8267b0b1-b622-489a-878d-6061b47da927\") " pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.346658 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.346631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6jjn\" (UniqueName: \"kubernetes.io/projected/43e86123-b87c-4297-acaa-ba1de2d293de-kube-api-access-k6jjn\") pod \"tuned-jqtw4\" (UID: \"43e86123-b87c-4297-acaa-ba1de2d293de\") " pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.347143 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.347001    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkpkf\" (UniqueName: \"kubernetes.io/projected/4b5333fe-432d-40d7-a893-35a95b16181f-kube-api-access-xkpkf\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.358746 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.358723    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-g9n22"
Jul 28 17:16:42.377572 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.377496    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-g9n22"
Jul 28 17:16:42.438408 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438373    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438422    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-cni-binary-copy\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-os-release\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438486    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-multus\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4s2\" (UniqueName: \"kubernetes.io/projected/02612ec5-b216-495d-92ab-3569ec0ee16e-kube-api-access-8l4s2\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438515    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438531    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-kubelet\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438579    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-system-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.438590 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-multus\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438602    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-hostroot\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-os-release\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438627    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-daemon-config\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-kubelet\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438664    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-bin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-system-cni-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438687    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-multus-certs\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-conf-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438717    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-var-lib-cni-bin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438716    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-hostroot\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-socket-dir-parent\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438755    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-multus-certs\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-k8s-cni-cncf-io\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-socket-dir-parent\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-conf-dir\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-etc-kubernetes\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-k8s-cni-cncf-io\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-netns\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-etc-kubernetes\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438883    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-cnibin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438928    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-cnibin\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.438927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/02612ec5-b216-495d-92ab-3569ec0ee16e-host-run-netns\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.439084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-cni-binary-copy\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.439748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.439084    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/02612ec5-b216-495d-92ab-3569ec0ee16e-multus-daemon-config\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.447138 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.447119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4s2\" (UniqueName: \"kubernetes.io/projected/02612ec5-b216-495d-92ab-3569ec0ee16e-kube-api-access-8l4s2\") pod \"multus-q24qv\" (UID: \"02612ec5-b216-495d-92ab-3569ec0ee16e\") " pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.517372 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.517323    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zxntr"
Jul 28 17:16:42.526018 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.525990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h"
Jul 28 17:16:42.534806 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.534780    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jqtw4"
Jul 28 17:16:42.540661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.540640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-g2rl4"
Jul 28 17:16:42.547394 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.547376    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-647fs"
Jul 28 17:16:42.554660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.554631    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:16:42.562381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.562362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-w5blq"
Jul 28 17:16:42.570794 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.570777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:16:42.576438 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.576418    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q24qv"
Jul 28 17:16:42.842052 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.842012    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:42.842052 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.842056    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:42.842282 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.842190    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:42.842282 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.842250    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:43.842234599 +0000 UTC m=+4.028827902 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:42.842385 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.842289    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:42.842385 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.842353    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:16:43.842337768 +0000 UTC m=+4.028931053 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:42.942994 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:42.942957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:42.943148 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.943104    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:42.943148 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.943137    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:42.943232 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.943150    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:42.943232 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:42.943207    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:43.943188677 +0000 UTC m=+4.129781960 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:43.220896 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.220873    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89932144_cfe2_4f4c_a545_64c463f6d879.slice/crio-fbb65da4a527d788b9430c0b411183da3e9e253c1ccd579ee65598a3f7eb76ea WatchSource:0}: Error finding container fbb65da4a527d788b9430c0b411183da3e9e253c1ccd579ee65598a3f7eb76ea: Status 404 returned error can't find the container with id fbb65da4a527d788b9430c0b411183da3e9e253c1ccd579ee65598a3f7eb76ea
Jul 28 17:16:43.221777 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.221748    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05ffd10_3b28_4ae6_9fd5_89b15f0f4139.slice/crio-8b2d2a1d8edbd1969b9200398648fbf3a45ecde74652199b3cf6479ca327d498 WatchSource:0}: Error finding container 8b2d2a1d8edbd1969b9200398648fbf3a45ecde74652199b3cf6479ca327d498: Status 404 returned error can't find the container with id 8b2d2a1d8edbd1969b9200398648fbf3a45ecde74652199b3cf6479ca327d498
Jul 28 17:16:43.222952 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.222929    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e86123_b87c_4297_acaa_ba1de2d293de.slice/crio-407a3dfbd5ccce0e5607287f0ca200d5a667e31de6079798945c29e6ad00a705 WatchSource:0}: Error finding container 407a3dfbd5ccce0e5607287f0ca200d5a667e31de6079798945c29e6ad00a705: Status 404 returned error can't find the container with id 407a3dfbd5ccce0e5607287f0ca200d5a667e31de6079798945c29e6ad00a705
Jul 28 17:16:43.224321 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.224288    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8267b0b1_b622_489a_878d_6061b47da927.slice/crio-195a53926488c659ce65c6cef815b200b577774b4266806e024adbad7863ae25 WatchSource:0}: Error finding container 195a53926488c659ce65c6cef815b200b577774b4266806e024adbad7863ae25: Status 404 returned error can't find the container with id 195a53926488c659ce65c6cef815b200b577774b4266806e024adbad7863ae25
Jul 28 17:16:43.225212 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.225183    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55868a3b_2011_4bda_8189_bdfdfebb3273.slice/crio-d884363982a200a5afaf0fab3899318b5096bc2ac3b46a0acea657fbebf5130d WatchSource:0}: Error finding container d884363982a200a5afaf0fab3899318b5096bc2ac3b46a0acea657fbebf5130d: Status 404 returned error can't find the container with id d884363982a200a5afaf0fab3899318b5096bc2ac3b46a0acea657fbebf5130d
Jul 28 17:16:43.226560 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:16:43.226470    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37aad0c_f620_44a7_9396_a8cdf7379c15.slice/crio-367bdf578116afcb55b1c4db69bf8d8273f8c04f2e71957de5226f9096eef414 WatchSource:0}: Error finding container 367bdf578116afcb55b1c4db69bf8d8273f8c04f2e71957de5226f9096eef414: Status 404 returned error can't find the container with id 367bdf578116afcb55b1c4db69bf8d8273f8c04f2e71957de5226f9096eef414
Jul 28 17:16:43.353109 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.352941    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:43.353243 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.352986    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:43.353243 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.353192    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:43.353362 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.353252    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:43.364038 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.364014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg46g" event={"ID":"df4df771-c44e-4a2f-a59a-f80f87f2e19c","Type":"ContainerStarted","Data":"945bdfd7733013f3d425b8ef6b409108f774dfcfa662b75c8bc7582cd1bf2a74"}
Jul 28 17:16:43.364975 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.364953    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-647fs" event={"ID":"8267b0b1-b622-489a-878d-6061b47da927","Type":"ContainerStarted","Data":"195a53926488c659ce65c6cef815b200b577774b4266806e024adbad7863ae25"}
Jul 28 17:16:43.366135 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.366109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5blq" event={"ID":"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139","Type":"ContainerStarted","Data":"8b2d2a1d8edbd1969b9200398648fbf3a45ecde74652199b3cf6479ca327d498"}
Jul 28 17:16:43.367788 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.367761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h" event={"ID":"89932144-cfe2-4f4c-a545-64c463f6d879","Type":"ContainerStarted","Data":"fbb65da4a527d788b9430c0b411183da3e9e253c1ccd579ee65598a3f7eb76ea"}
Jul 28 17:16:43.369452 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.369429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerStarted","Data":"ca23276227309db30e1668e2de7f3f4735b48e1a887cfdcbd0f71b238363a8ae"}
Jul 28 17:16:43.371597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.371573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" event={"ID":"f0f6325ce07165ebf2e9979367b54cf4","Type":"ContainerStarted","Data":"07ef3fff5f8fb61b57f89d1d5405a829a9cff49bca07809c5567750f649fa205"}
Jul 28 17:16:43.372822 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.372794    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"367bdf578116afcb55b1c4db69bf8d8273f8c04f2e71957de5226f9096eef414"}
Jul 28 17:16:43.373900 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.373881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxntr" event={"ID":"55868a3b-2011-4bda-8189-bdfdfebb3273","Type":"ContainerStarted","Data":"d884363982a200a5afaf0fab3899318b5096bc2ac3b46a0acea657fbebf5130d"}
Jul 28 17:16:43.374805 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.374787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jqtw4" event={"ID":"43e86123-b87c-4297-acaa-ba1de2d293de","Type":"ContainerStarted","Data":"407a3dfbd5ccce0e5607287f0ca200d5a667e31de6079798945c29e6ad00a705"}
Jul 28 17:16:43.375827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.375808    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q24qv" event={"ID":"02612ec5-b216-495d-92ab-3569ec0ee16e","Type":"ContainerStarted","Data":"c80fbe531dcdb9a0944d32bb547592a745afe9aafa68ef50be03b7ea2f462ff6"}
Jul 28 17:16:43.380900 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.380871    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:42 +0000 UTC" deadline="2028-04-04 09:11:42.527867302 +0000 UTC"
Jul 28 17:16:43.380900 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.380900    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14775h54m59.146971777s"
Jul 28 17:16:43.387951 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.387910    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-134-28.ec2.internal" podStartSLOduration=2.387898125 podStartE2EDuration="2.387898125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:43.387445941 +0000 UTC m=+3.574039247" watchObservedRunningTime="2026-07-28 17:16:43.387898125 +0000 UTC m=+3.574491430"
Jul 28 17:16:43.850331 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.850291    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:43.850499 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.850385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:43.850611 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.850520    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:43.850611 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.850604    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.850585217 +0000 UTC m=+6.037178513 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:43.851001 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.850985    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:43.851070 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.851039    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.851024213 +0000 UTC m=+6.037617510 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:43.953197 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:43.952560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:43.953197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.952768    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:43.953197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.952789    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:43.953197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.952802    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:43.953197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:43.952861    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:45.952844654 +0000 UTC m=+6.139437940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:44.354930 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:44.354421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:44.354930 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:44.354561    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:44.381489 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:44.381444    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:11:42 +0000 UTC" deadline="2028-01-09 02:07:44.036717696 +0000 UTC"
Jul 28 17:16:44.381489 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:44.381487    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12704h50m59.655235067s"
Jul 28 17:16:44.392179 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:44.391645    2576 generic.go:356] "Generic (PLEG): container finished" podID="0f8670d84f658402c20684d057cb8783" containerID="88901d65014776eb0598e1006b6a20065b7ca642a437327ef196a5be68dc4834" exitCode=0
Jul 28 17:16:44.392179 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:44.391835    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"0f8670d84f658402c20684d057cb8783","Type":"ContainerDied","Data":"88901d65014776eb0598e1006b6a20065b7ca642a437327ef196a5be68dc4834"}
Jul 28 17:16:45.352892 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.352819    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:45.353041 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.352952    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:45.353338 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.353319    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:45.353452 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.353432    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:45.413898 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.413863    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" event={"ID":"0f8670d84f658402c20684d057cb8783","Type":"ContainerStarted","Data":"e740ca1bfada41510477a0fb554eb72a8193e0b6719d057d1dc96903adaad1f6"}
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.875334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.875418    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.875564    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.875626    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:16:49.875606521 +0000 UTC m=+10.062199806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.875694    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.876067 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.875730    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:49.875719255 +0000 UTC m=+10.062312539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:45.976649 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:45.976020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:45.976649 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.976221    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:45.976649 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.976239    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:45.976649 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.976252    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:45.976649 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:45.976305    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:49.976287737 +0000 UTC m=+10.162881038 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:46.354285 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:46.353822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:46.354285 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:46.353947    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:47.352568 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:47.352476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:47.353051 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:47.352604    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:47.353051 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:47.352783    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:47.353051 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:47.352893    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:48.354050 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:48.353539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:48.354050 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:48.353697    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:49.353329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:49.353169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:49.353329 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.353309    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:49.353631 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:49.353608    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:49.353747 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.353727    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:49.909103 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:49.909064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:49.909573 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:49.909132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:49.909573 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.909267    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:49.909573 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.909326    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:57.909307957 +0000 UTC m=+18.095901257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:49.909756 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.909715    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:49.909807 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:49.909759    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:16:57.909745162 +0000 UTC m=+18.096338451 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:50.010216 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:50.010172    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:50.010407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:50.010319    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:50.010407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:50.010341    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:50.010407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:50.010353    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:50.010407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:50.010405    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:16:58.010390611 +0000 UTC m=+18.196983909 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:50.354280 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:50.354247    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:50.354457 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:50.354369    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:51.353749 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:51.353390    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:51.353749 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:51.353510    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:51.353749 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:51.353662    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:51.354275 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:51.353782    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:52.353216 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:52.353181    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:52.353394 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:52.353302    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:53.353268 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:53.353236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:53.353745 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:53.353276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:53.353745 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:53.353367    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:53.353745 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:53.353473    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:54.352972 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:54.352891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:54.353178 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:54.353015    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:55.353379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:55.353348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:55.353815 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:55.353492    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:55.353815 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:55.353555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:55.353815 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:55.353653    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:56.353471 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:56.353438    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:56.353957 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:56.353564    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:57.352666 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:57.352629    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:57.352878 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.352749    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:16:57.352878 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:57.352805    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:57.352988 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.352923    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:57.970455 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:57.970411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:57.970936 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:57.970480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:57.970936 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.970621    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:57.970936 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.970631    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:57.970936 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.970696    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:13.970677438 +0000 UTC m=+34.157270947 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:16:57.970936 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:57.970717    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:13.97070793 +0000 UTC m=+34.157301213 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:16:58.071556 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:58.071514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:58.071730 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:58.071676    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:16:58.071730 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:58.071699    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:16:58.071730 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:58.071710    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:58.071882 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:58.071773    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:14.071754972 +0000 UTC m=+34.258348259 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:16:58.352555 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:58.352519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:16:58.352714 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:58.352665    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:16:59.353007 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:59.352968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:16:59.353398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:16:59.352968    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:16:59.353398 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:59.353108    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:16:59.353398 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:16:59.353142    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:00.356892 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:00.355378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:00.356892 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:00.355617    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:01.353382 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:01.353328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:01.353634 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:01.353328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:01.353634 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:01.353465    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:01.353634 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:01.353539    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:02.353745 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.353296    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:02.353745 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:02.353714    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:02.451723 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.451691    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="ed58b9a125b93df8eb8d3ae9349f2fc04a8b27e816dbc62df6d4898c3258e8a3" exitCode=0
Jul 28 17:17:02.451845 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.451775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"ed58b9a125b93df8eb8d3ae9349f2fc04a8b27e816dbc62df6d4898c3258e8a3"}
Jul 28 17:17:02.455266 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455233    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"6ae25c1a96f2d6924b7d9ffde924ac2be073fc322641284016654ca55c3153b2"}
Jul 28 17:17:02.455362 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455311    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"7a5ef9277c039b5be37eb4fc5b14e974bb282c74f4d4079607c082028f12ee6b"}
Jul 28 17:17:02.455362 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"f4230f621d2ab7e91ad4f6ec557cc7bc2ecc099ce86746e541e0a75a6092fa2a"}
Jul 28 17:17:02.455461 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455393    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"2a5e7b43de8c270a15b0cec8f05ad68badf4a9bb44ef36bce3539b16b43470d1"}
Jul 28 17:17:02.455461 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"071da2c4b95ca3979d0971f742e5cc9abb04b134c3e0249bea8df55685e45762"}
Jul 28 17:17:02.455573 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.455465    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"88822750c9a9d6a1f51206882bb42dde021bb222dbfb2946cf20b9a6181f08ed"}
Jul 28 17:17:02.457003 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.456977    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jqtw4" event={"ID":"43e86123-b87c-4297-acaa-ba1de2d293de","Type":"ContainerStarted","Data":"acae58e05b8edc1682cd5537636310f2d19912133a952e9966087440b04b801f"}
Jul 28 17:17:02.458763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.458696    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q24qv" event={"ID":"02612ec5-b216-495d-92ab-3569ec0ee16e","Type":"ContainerStarted","Data":"00a959206f606db9341f05e76570c102358ecea13977a55dca73a93494c84d5c"}
Jul 28 17:17:02.460300 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.460279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qg46g" event={"ID":"df4df771-c44e-4a2f-a59a-f80f87f2e19c","Type":"ContainerStarted","Data":"02a946c522264ded282f1f45b54925405cc365cb556447a3eb2a5a3b7463df6b"}
Jul 28 17:17:02.461859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.461832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-647fs" event={"ID":"8267b0b1-b622-489a-878d-6061b47da927","Type":"ContainerStarted","Data":"3a5ad6116617d2f0d1679ef8ad73ee33a6b7f470e87c4a8d6062cdc2810aae1f"}
Jul 28 17:17:02.463212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.463186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-w5blq" event={"ID":"b05ffd10-3b28-4ae6-9fd5-89b15f0f4139","Type":"ContainerStarted","Data":"f2b549a312bbe167b2f0c84d41d8b12921a08cb828cb5efb0826f3c6d1888080"}
Jul 28 17:17:02.464482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.464463    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h" event={"ID":"89932144-cfe2-4f4c-a545-64c463f6d879","Type":"ContainerStarted","Data":"05b12902878a2237be082844933a9f1756f4cd0cc61381ec74ab6418e3bdede2"}
Jul 28 17:17:02.482142 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.482097    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-134-28.ec2.internal" podStartSLOduration=21.482083325 podStartE2EDuration="21.482083325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:16:45.436319647 +0000 UTC m=+5.622912953" watchObservedRunningTime="2026-07-28 17:17:02.482083325 +0000 UTC m=+22.668676630"
Jul 28 17:17:02.497356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.497314    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-w5blq" podStartSLOduration=4.022333503 podStartE2EDuration="22.497293153s" totalImagesPullingTime="18.47495965s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.497213617 +0000 UTC m=+22.683806918" watchObservedRunningTime="2026-07-28 17:17:02.497293153 +0000 UTC m=+22.683886458"
Jul 28 17:17:02.514610 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.514527    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q24qv" podStartSLOduration=3.987145748 podStartE2EDuration="22.514513914s" totalImagesPullingTime="18.527368166s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.514268068 +0000 UTC m=+22.700861376" watchObservedRunningTime="2026-07-28 17:17:02.514513914 +0000 UTC m=+22.701107218"
Jul 28 17:17:02.532153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.532086    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jqtw4" podStartSLOduration=4.05787979 podStartE2EDuration="22.532071282s" totalImagesPullingTime="18.474191492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.531733486 +0000 UTC m=+22.718326792" watchObservedRunningTime="2026-07-28 17:17:02.532071282 +0000 UTC m=+22.718664590"
Jul 28 17:17:02.548229 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.548193    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-647fs" podStartSLOduration=4.076018671 podStartE2EDuration="22.548180226s" totalImagesPullingTime="18.472161555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.547769827 +0000 UTC m=+22.734363133" watchObservedRunningTime="2026-07-28 17:17:02.548180226 +0000 UTC m=+22.734773528"
Jul 28 17:17:02.567609 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:02.567569    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qg46g" podStartSLOduration=8.297867994 podStartE2EDuration="22.56753826s" totalImagesPullingTime="14.269670266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:02.567420373 +0000 UTC m=+22.754013679" watchObservedRunningTime="2026-07-28 17:17:02.56753826 +0000 UTC m=+22.754131581"
Jul 28 17:17:03.122171 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.122148    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:03.292444 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.292335    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:03.122168067Z","UUID":"f6901d17-e4a6-4b8e-b955-b8b1a0f5e9ad","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:03.294032 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.294011    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:03.294156 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.294056    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:03.353018 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.352988    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:03.353182 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.352987    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:03.353182 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:03.353091    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:03.353296 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:03.353190    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:03.469130 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.469050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h" event={"ID":"89932144-cfe2-4f4c-a545-64c463f6d879","Type":"ContainerStarted","Data":"678a105f5f8f6e321e06973ac6cb08cff7ee855999ba68d9b55b1be11bfcf657"}
Jul 28 17:17:03.470769 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.470694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zxntr" event={"ID":"55868a3b-2011-4bda-8189-bdfdfebb3273","Type":"ContainerStarted","Data":"ff6bff7235e5ec5215c73be88a08f469e0cefdb12d0ae88f1dfddd404e7bcda2"}
Jul 28 17:17:03.487815 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.487768    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zxntr" podStartSLOduration=5.016830274 podStartE2EDuration="23.487749852s" totalImagesPullingTime="18.470919578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:03.487196396 +0000 UTC m=+23.673789702" watchObservedRunningTime="2026-07-28 17:17:03.487749852 +0000 UTC m=+23.674343169"
Jul 28 17:17:03.997654 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.997618    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:17:03.998724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:03.998690    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:17:04.352767 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:04.352736    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:04.352944 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:04.352871    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:04.473171 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:04.473115    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:17:04.473662 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:04.473643    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qg46g"
Jul 28 17:17:05.353663 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:05.353441    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:05.353967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:05.353503    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:05.353967 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:05.353750    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:05.353967 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:05.353854    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:05.478163 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:05.478128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h" event={"ID":"89932144-cfe2-4f4c-a545-64c463f6d879","Type":"ContainerStarted","Data":"adcbe39cdb59ae14a740772f3da4404160a6df427a4d35555838cf34f897f539"}
Jul 28 17:17:05.482060 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:05.482031    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"7df10d43f6e7f16124ea0b81adc309bf9d1153af2f9c72e2dc06d544078524e2"}
Jul 28 17:17:05.498823 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:05.498773    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ttg7h" podStartSLOduration=4.426327128 podStartE2EDuration="25.498756027s" totalImagesPullingTime="21.072428899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:05.498274766 +0000 UTC m=+25.684868072" watchObservedRunningTime="2026-07-28 17:17:05.498756027 +0000 UTC m=+25.685349336"
Jul 28 17:17:06.353451 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:06.353421    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:06.353647 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:06.353562    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:07.352649 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.352463    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:07.353289 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.352475    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:07.353289 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:07.352725    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:07.353289 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:07.352803    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:07.489490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.489459    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" event={"ID":"d37aad0c-f620-44a7-9396-a8cdf7379c15","Type":"ContainerStarted","Data":"13de34bfbe12dd5115e5ffb3929f506f260cc68f4d6c8c474243a170840fff04"}
Jul 28 17:17:07.489695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.489673    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:07.491363 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.491338    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="d3ea2bd7ecb7a48af55b73309fdd47719ce614270d9fc5da93adb60ab29dcd4b" exitCode=0
Jul 28 17:17:07.491459 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.491372    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"d3ea2bd7ecb7a48af55b73309fdd47719ce614270d9fc5da93adb60ab29dcd4b"}
Jul 28 17:17:07.505484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.505464    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:07.521049 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:07.521003    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5" podStartSLOduration=8.979912597 podStartE2EDuration="27.520990316s" totalImagesPullingTime="18.541077719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:07.520674119 +0000 UTC m=+27.707267423" watchObservedRunningTime="2026-07-28 17:17:07.520990316 +0000 UTC m=+27.707583621"
Jul 28 17:17:08.352772 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:08.352742    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:08.353209 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:08.352842    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:08.493812 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:08.493786    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:08.493996 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:08.493851    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:08.508298 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:08.508276    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:09.179389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.179361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqq8n"]
Jul 28 17:17:09.179632 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.179562    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:09.179709 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:09.179688    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:09.181897 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.181873    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5swwd"]
Jul 28 17:17:09.182014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.181972    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:09.182085 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:09.182067    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:09.182331 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.182309    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5p9w"]
Jul 28 17:17:09.182403 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.182393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:09.182488 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:09.182463    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:09.498097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.498024    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="a7143895008038b15d05cd6f864b35f61c00bb121dc62c88d9e8135805de76e9" exitCode=0
Jul 28 17:17:09.498463 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:09.498112    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"a7143895008038b15d05cd6f864b35f61c00bb121dc62c88d9e8135805de76e9"}
Jul 28 17:17:10.354024 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:10.353993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:10.354213 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:10.354101    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:11.352716 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:11.352522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:11.353248 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:11.352522    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:11.353248 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:11.352797    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:11.353248 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:11.352869    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:11.504022 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:11.503990    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="f8e73c3f97b2c694d68dcf53dd4bf8cc9811eb201120d58a6c0fb59ff7bdc7c4" exitCode=0
Jul 28 17:17:11.504169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:11.504051    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"f8e73c3f97b2c694d68dcf53dd4bf8cc9811eb201120d58a6c0fb59ff7bdc7c4"}
Jul 28 17:17:12.352690 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:12.352658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:12.352861 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:12.352762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:13.353096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:13.353060    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:13.353096 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:13.353096    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:13.353594 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.353185    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqq8n" podUID="4b5333fe-432d-40d7-a893-35a95b16181f"
Jul 28 17:17:13.353594 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.353372    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5swwd" podUID="9a6a32c3-2640-4cf8-8a4f-daf938694431"
Jul 28 17:17:13.995087 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:13.995051    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:13.995262 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:13.995112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:13.995262 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.995228    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:13.995376 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.995306    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.995285725 +0000 UTC m=+66.181879008 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:13.995376 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.995235    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:13.995478 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:13.995400    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret podName:9a6a32c3-2640-4cf8-8a4f-daf938694431 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:45.995375213 +0000 UTC m=+66.181968500 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret") pod "global-pull-secret-syncer-5swwd" (UID: "9a6a32c3-2640-4cf8-8a4f-daf938694431") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:14.095485 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.095446    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:14.095686 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.095659    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:14.095756 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.095687    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:14.095756 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.095699    2576 projected.go:196] Error preparing data for projected volume kube-api-access-xflrk for pod openshift-network-diagnostics/network-check-target-k5p9w: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:14.095847 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.095757    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk podName:8eb356a4-039c-4206-ac7c-fae1b3a938f3 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.095741276 +0000 UTC m=+66.282334558 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-xflrk" (UniqueName: "kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk") pod "network-check-target-k5p9w" (UID: "8eb356a4-039c-4206-ac7c-fae1b3a938f3") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:14.353339 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.353307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:14.353930 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.353440    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-k5p9w" podUID="8eb356a4-039c-4206-ac7c-fae1b3a938f3"
Jul 28 17:17:14.633829 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.633750    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-134-28.ec2.internal" event="NodeReady"
Jul 28 17:17:14.633982 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.633891    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:17:14.678110 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.678075    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cc55c59d-spwbb"]
Jul 28 17:17:14.680933 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.680910    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.685141 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.684921    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:17:14.685141 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.684978    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:17:14.685141 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.684996    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:17:14.685141 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.685048    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4mv5p\""
Jul 28 17:17:14.691083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.690984    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:17:14.691427 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.691403    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwds"]
Jul 28 17:17:14.693497 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.693451    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cc55c59d-spwbb"]
Jul 28 17:17:14.693618 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.693563    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.693691 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.693623    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jnqnd"]
Jul 28 17:17:14.695610 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.695594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.696261 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.696242    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:17:14.696636 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.696608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-wqh9j\""
Jul 28 17:17:14.696636 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.696617    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:17:14.696636 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.696630    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:17:14.698006 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.697990    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:17:14.698385 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.698366    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:17:14.698485 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.698453    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjb4d\""
Jul 28 17:17:14.701595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.701573    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwds"]
Jul 28 17:17:14.707110 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.707092    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnqnd"]
Jul 28 17:17:14.801746 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.801919 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.801919 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/988ee349-bc90-4a64-9787-a4b7688e5f19-config-volume\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.801919 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/988ee349-bc90-4a64-9787-a4b7688e5f19-tmp-dir\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.801919 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801893    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802080 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.801950    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldmz6\" (UniqueName: \"kubernetes.io/projected/e434feb5-2d72-4980-a231-12e3ca1449c2-kube-api-access-ldmz6\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.802080 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802013    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802080 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802029    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tt48\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802080 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802080 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802068    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e434feb5-2d72-4980-a231-12e3ca1449c2-cert\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.802288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xd7g\" (UniqueName: \"kubernetes.io/projected/988ee349-bc90-4a64-9787-a4b7688e5f19-kube-api-access-6xd7g\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.802288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802163    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802218    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.802288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.802246    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903131 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldmz6\" (UniqueName: \"kubernetes.io/projected/e434feb5-2d72-4980-a231-12e3ca1449c2-kube-api-access-ldmz6\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.903131 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903109    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tt48\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903208    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e434feb5-2d72-4980-a231-12e3ca1449c2-cert\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.903352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xd7g\" (UniqueName: \"kubernetes.io/projected/988ee349-bc90-4a64-9787-a4b7688e5f19-kube-api-access-6xd7g\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.903352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903371    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903400    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903447    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/988ee349-bc90-4a64-9787-a4b7688e5f19-config-volume\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903563    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/988ee349-bc90-4a64-9787-a4b7688e5f19-tmp-dir\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.903651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.903696    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.903717    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.903785    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.403764218 +0000 UTC m=+35.590357501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.903909    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:14.903969    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:15.403950923 +0000 UTC m=+35.590544212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:14.904000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.903991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.904302 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.904243    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/988ee349-bc90-4a64-9787-a4b7688e5f19-config-volume\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.904398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.904378    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/988ee349-bc90-4a64-9787-a4b7688e5f19-tmp-dir\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.904673 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.904631    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.904906 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.904885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.908081 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.908057    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.908164 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.908119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.908207 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.908184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e434feb5-2d72-4980-a231-12e3ca1449c2-cert\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.915254 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.915196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tt48\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:14.915254 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.915196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xd7g\" (UniqueName: \"kubernetes.io/projected/988ee349-bc90-4a64-9787-a4b7688e5f19-kube-api-access-6xd7g\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:14.915507 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.915467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldmz6\" (UniqueName: \"kubernetes.io/projected/e434feb5-2d72-4980-a231-12e3ca1449c2-kube-api-access-ldmz6\") pod \"ingress-canary-mrwds\" (UID: \"e434feb5-2d72-4980-a231-12e3ca1449c2\") " pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:14.915769 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:14.915703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:15.005247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.005211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mrwds"
Jul 28 17:17:15.352807 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.352777    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:15.352995 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.352782    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:15.355901 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.355880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:15.357027 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.357005    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvwzr\""
Jul 28 17:17:15.357169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.357058    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:15.407070 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.406886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:15.407213 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.407094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:15.407213 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:15.407034    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:15.407213 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:15.407203    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:15.407213 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:15.407214    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:15.407412 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:15.407205    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.407186803 +0000 UTC m=+36.593780085 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:15.407412 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:15.407269    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:16.407255981 +0000 UTC m=+36.593849272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:15.527859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:15.527836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mrwds"]
Jul 28 17:17:15.531187 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:17:15.531145    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode434feb5_2d72_4980_a231_12e3ca1449c2.slice/crio-55bcee349b96b21c2ffa2eeb466575ebd9081901f436c70780fd68980e6ab978 WatchSource:0}: Error finding container 55bcee349b96b21c2ffa2eeb466575ebd9081901f436c70780fd68980e6ab978: Status 404 returned error can't find the container with id 55bcee349b96b21c2ffa2eeb466575ebd9081901f436c70780fd68980e6ab978
Jul 28 17:17:16.353385 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.353348    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:16.356784 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.356764    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:17:16.357983 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.357964    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qzphg\""
Jul 28 17:17:16.357983 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.357978    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:17:16.416764 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.416740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:16.416900 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.416810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:16.416952 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:16.416902    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:16.416952 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:16.416911    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:16.416952 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:16.416931    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:16.417090 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:16.416955    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:18.416936636 +0000 UTC m=+38.603529918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:16.417090 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:16.416981    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:18.416963877 +0000 UTC m=+38.603557163 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:16.517994 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.517964    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="be59b6f2b6c7131caa74e9b751114efd0e18c72a9c4fd6bd2afed140f1ae5647" exitCode=0
Jul 28 17:17:16.518146 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.518030    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"be59b6f2b6c7131caa74e9b751114efd0e18c72a9c4fd6bd2afed140f1ae5647"}
Jul 28 17:17:16.519316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:16.519143    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrwds" event={"ID":"e434feb5-2d72-4980-a231-12e3ca1449c2","Type":"ContainerStarted","Data":"55bcee349b96b21c2ffa2eeb466575ebd9081901f436c70780fd68980e6ab978"}
Jul 28 17:17:17.524978 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:17.524946    2576 generic.go:356] "Generic (PLEG): container finished" podID="07acdd3f-1248-4bb5-ab18-ad1cdf17f694" containerID="4cbb3680b0866cb9afb192b3a7eeaecb53516d216196671654ea84c4919723fe" exitCode=0
Jul 28 17:17:17.525401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:17.525014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerDied","Data":"4cbb3680b0866cb9afb192b3a7eeaecb53516d216196671654ea84c4919723fe"}
Jul 28 17:17:18.429591 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:18.429316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:18.429714 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:18.429645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:18.429825 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:18.429452    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:18.429825 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:18.429698    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:18.429825 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:18.429781    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:18.429825 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:18.429795    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:22.429769061 +0000 UTC m=+42.616362364 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:18.429825 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:18.429818    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:22.429807599 +0000 UTC m=+42.616400887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:18.529787 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:18.529754    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" event={"ID":"07acdd3f-1248-4bb5-ab18-ad1cdf17f694","Type":"ContainerStarted","Data":"aadfc192880d31736b7368bc9d2c7f0f4945d877c81500beb90303ff5170a64d"}
Jul 28 17:17:18.560024 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:18.559972    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-g2rl4" podStartSLOduration=9.570741439 podStartE2EDuration="38.559952758s" totalImagesPullingTime="28.989211319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:18.55942636 +0000 UTC m=+38.746019657" watchObservedRunningTime="2026-07-28 17:17:18.559952758 +0000 UTC m=+38.746546064"
Jul 28 17:17:18.821226 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:18.821137    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:19.533814 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:19.533781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mrwds" event={"ID":"e434feb5-2d72-4980-a231-12e3ca1449c2","Type":"ContainerStarted","Data":"fcaeff1a78dcd28128fc360d4b523d89c4441c00d0f7b29843fa3089a26c4334"}
Jul 28 17:17:19.553877 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:19.553831    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mrwds" podStartSLOduration=2.493825608 podStartE2EDuration="5.553816988s" totalImagesPullingTime="3.05999138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:19.552362911 +0000 UTC m=+39.738956217" watchObservedRunningTime="2026-07-28 17:17:19.553816988 +0000 UTC m=+39.740410314"
Jul 28 17:17:20.677170 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:20.677141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.458020 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:22.457984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:22.458060    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:22.458130    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:22.458148    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:22.458150    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:22.458198    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:30.458185189 +0000 UTC m=+50.644778472 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:22.458467 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:22.458211    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:30.458205139 +0000 UTC m=+50.644798422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:22.559971 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:22.559935    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.855367 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:22.855333    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:22.940871 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:22.940833    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.036102 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:23.036077    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.582199 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:23.582166    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.682991 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:23.682962    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:23.977991 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:23.977910    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.079432 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:24.079400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.783976 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:24.783945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:24.892843 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:24.892809    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.174122 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:25.174034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.258743 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:25.258712    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:25.977008 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:25.976947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:26.076975 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:26.076944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:26.382563 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:26.382512    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:26.483555 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:26.483519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:27.230061 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:27.230024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:27.572563 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:27.572516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:27.673760 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:27.673731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:29.595161 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:29.595127    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:29.859310 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:29.859230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:30.370027 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:30.369998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:30.518813 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:30.518777    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:30.518982 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:30.518833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:30.518982 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:30.518930    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:30.518982 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:30.518942    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:30.518982 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:30.518953    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:30.519104 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:30.519004    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.518984208 +0000 UTC m=+66.705577494 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:30.519104 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:30.519024    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.51901557 +0000 UTC m=+66.705608858 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:40.519610 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:40.519576    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-vcdd5"
Jul 28 17:17:40.783360 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:40.783283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:40.883085 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:40.883055    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.472181 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:41.472145    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:41.573602 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:41.573561    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:42.469398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:42.469347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:42.535524 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:42.535493    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:42.606083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:42.606059    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.474758 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:43.474722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.554462 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:43.554431    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.655209 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:43.655183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:43.755771 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:43.755715    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:46.025553 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.025484    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:46.026020 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.025593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:17:46.028877 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.028855    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:46.029009 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.028887    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:46.036474 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.036455    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 28 17:17:46.036528 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.036520    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs podName:4b5333fe-432d-40d7-a893-35a95b16181f nodeName:}" failed. No retries permitted until 2026-07-28 17:18:50.036505171 +0000 UTC m=+130.223098458 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs") pod "network-metrics-daemon-tqq8n" (UID: "4b5333fe-432d-40d7-a893-35a95b16181f") : secret "metrics-daemon-secret" not found
Jul 28 17:17:46.040134 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.040113    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/9a6a32c3-2640-4cf8-8a4f-daf938694431-original-pull-secret\") pod \"global-pull-secret-syncer-5swwd\" (UID: \"9a6a32c3-2640-4cf8-8a4f-daf938694431\") " pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:46.126837 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.126802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:46.129981 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.129963    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:17:46.140586 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.140567    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:17:46.150065 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.150040    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflrk\" (UniqueName: \"kubernetes.io/projected/8eb356a4-039c-4206-ac7c-fae1b3a938f3-kube-api-access-xflrk\") pod \"network-check-target-k5p9w\" (UID: \"8eb356a4-039c-4206-ac7c-fae1b3a938f3\") " pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:46.265260 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.265230    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5swwd"
Jul 28 17:17:46.365223 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.365197    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-qzphg\""
Jul 28 17:17:46.373366 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.373343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:46.385647 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.385623    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5swwd"]
Jul 28 17:17:46.388696 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:17:46.388669    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a6a32c3_2640_4cf8_8a4f_daf938694431.slice/crio-92d7cca3418527cf37e8f7de55472f75639756c9d6a11dc35a5cb4b8757aaf1f WatchSource:0}: Error finding container 92d7cca3418527cf37e8f7de55472f75639756c9d6a11dc35a5cb4b8757aaf1f: Status 404 returned error can't find the container with id 92d7cca3418527cf37e8f7de55472f75639756c9d6a11dc35a5cb4b8757aaf1f
Jul 28 17:17:46.502757 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.502723    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-k5p9w"]
Jul 28 17:17:46.506505 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:17:46.506485    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb356a4_039c_4206_ac7c_fae1b3a938f3.slice/crio-ed5035b0e647bd07db7a42e60ded014dada37bd8c3295c61ef597324e6be5c5e WatchSource:0}: Error finding container ed5035b0e647bd07db7a42e60ded014dada37bd8c3295c61ef597324e6be5c5e: Status 404 returned error can't find the container with id ed5035b0e647bd07db7a42e60ded014dada37bd8c3295c61ef597324e6be5c5e
Jul 28 17:17:46.533370 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.533348    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:17:46.533470 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.533397    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:17:46.533509 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.533482    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:46.533509 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.533486    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:46.533509 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.533503    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:17:46.533617 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.533536    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls podName:988ee349-bc90-4a64-9787-a4b7688e5f19 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:18.533523249 +0000 UTC m=+98.720116532 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls") pod "dns-default-jnqnd" (UID: "988ee349-bc90-4a64-9787-a4b7688e5f19") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:46.533617 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:46.533564    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:18.53355805 +0000 UTC m=+98.720151334 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:17:46.595529 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.595462    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5p9w" event={"ID":"8eb356a4-039c-4206-ac7c-fae1b3a938f3","Type":"ContainerStarted","Data":"ed5035b0e647bd07db7a42e60ded014dada37bd8c3295c61ef597324e6be5c5e"}
Jul 28 17:17:46.596409 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:46.596386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5swwd" event={"ID":"9a6a32c3-2640-4cf8-8a4f-daf938694431","Type":"ContainerStarted","Data":"92d7cca3418527cf37e8f7de55472f75639756c9d6a11dc35a5cb4b8757aaf1f"}
Jul 28 17:17:51.371895 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.371868    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-84996d74f-sgkml"]
Jul 28 17:17:51.373743 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.373725    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.377484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377278    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 28 17:17:51.377484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 28 17:17:51.377484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377331    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4vgcq\""
Jul 28 17:17:51.377716 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377558    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 28 17:17:51.377774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377731    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 28 17:17:51.377940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.377923    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 28 17:17:51.378531 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.378517    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 28 17:17:51.391654 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.391630    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84996d74f-sgkml"]
Jul 28 17:17:51.470035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.469960    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.470035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.470001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-stats-auth\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.470035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.470019    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.470326 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.470101    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-default-certificate\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.470326 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.470179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf62w\" (UniqueName: \"kubernetes.io/projected/0d3cc9dd-9a09-4852-8f44-50c87d93c575-kube-api-access-qf62w\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571296 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.571197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571296 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.571236    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-stats-auth\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571296 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.571254    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571296 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.571283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-default-certificate\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571633 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:51.571368    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:51.571633 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:51.571373    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.071354879 +0000 UTC m=+72.257948161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:51.571633 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.571458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf62w\" (UniqueName: \"kubernetes.io/projected/0d3cc9dd-9a09-4852-8f44-50c87d93c575-kube-api-access-qf62w\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.571633 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:51.571500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:52.071483052 +0000 UTC m=+72.258076350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:17:51.573622 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.573604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-default-certificate\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.573794 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.573775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-stats-auth\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.581015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.580986    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf62w\" (UniqueName: \"kubernetes.io/projected/0d3cc9dd-9a09-4852-8f44-50c87d93c575-kube-api-access-qf62w\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:51.610315 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.610290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-k5p9w" event={"ID":"8eb356a4-039c-4206-ac7c-fae1b3a938f3","Type":"ContainerStarted","Data":"599d0de29e1766c09c0e6d8a944ad838b2f994554fcbf629566a7fff4381c173"}
Jul 28 17:17:51.610450 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.610423    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:17:51.611802 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.611780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5swwd" event={"ID":"9a6a32c3-2640-4cf8-8a4f-daf938694431","Type":"ContainerStarted","Data":"7eaa912934f00ae5cb5e4e27ee05bb2322a8e544e90d5f2197ccd338ba2412e3"}
Jul 28 17:17:51.630373 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:51.630328    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-k5p9w" podStartSLOduration=66.848302794 podStartE2EDuration="1m11.630316236s" totalImagesPullingTime="4.782013442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:51.629855219 +0000 UTC m=+71.816448525" watchObservedRunningTime="2026-07-28 17:17:51.630316236 +0000 UTC m=+71.816909538"
Jul 28 17:17:52.075101 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:52.075074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:52.075101 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:52.075107    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:52.075301 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:52.075255    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:53.075236502 +0000 UTC m=+73.261829802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:52.075301 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:52.075275    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:52.075372 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:52.075311    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:53.075301151 +0000 UTC m=+73.261894447 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:17:53.081406 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:53.081364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:53.081406 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:53.081408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:53.081833 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:53.081531    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:53.081833 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:53.081536    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:55.081518657 +0000 UTC m=+75.268111940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:53.081833 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:53.081580    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:55.081569379 +0000 UTC m=+75.268162662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:17:54.724654 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.724595    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5swwd" podStartSLOduration=69.821390365 podStartE2EDuration="1m14.724577495s" totalImagesPullingTime="4.90318713s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:51.649966337 +0000 UTC m=+71.836559644" watchObservedRunningTime="2026-07-28 17:17:54.724577495 +0000 UTC m=+74.911170799"
Jul 28 17:17:54.725093 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.725078    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f7nmc"]
Jul 28 17:17:54.729622 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.729592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.732202 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.732175    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:17:54.732316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.732175    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:17:54.732316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.732184    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:17:54.732316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.732184    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:17:54.733506 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.733490    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-mc5b9\""
Jul 28 17:17:54.739173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.739149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f7nmc"]
Jul 28 17:17:54.794136 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.794109    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465c3991-57e5-4a79-8a94-78763ce4180e-data-volume\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.794232 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.794149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465c3991-57e5-4a79-8a94-78763ce4180e-crio-socket\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.794232 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.794183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.794232 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.794197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrth6\" (UniqueName: \"kubernetes.io/projected/465c3991-57e5-4a79-8a94-78763ce4180e-kube-api-access-zrth6\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.794445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.794296    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.894773 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.894744    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.894859 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.894776    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrth6\" (UniqueName: \"kubernetes.io/projected/465c3991-57e5-4a79-8a94-78763ce4180e-kube-api-access-zrth6\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.894904 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.894858    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.894904 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.894892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465c3991-57e5-4a79-8a94-78763ce4180e-data-volume\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.895001 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:54.894860    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:54.895001 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.894933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465c3991-57e5-4a79-8a94-78763ce4180e-crio-socket\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.895001 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:54.894961    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:54.895114 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:54.895008    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:55.394988185 +0000 UTC m=+75.581581472 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:54.895114 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:54.895030    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:55.395020074 +0000 UTC m=+75.581613364 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:54.895114 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.895026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/465c3991-57e5-4a79-8a94-78763ce4180e-crio-socket\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.895280 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.895262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/465c3991-57e5-4a79-8a94-78763ce4180e-data-volume\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:54.906122 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:54.906097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrth6\" (UniqueName: \"kubernetes.io/projected/465c3991-57e5-4a79-8a94-78763ce4180e-kube-api-access-zrth6\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:55.096000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:55.095962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:55.096000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:55.096000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:55.096205 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.096103    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:55.096205 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.096140    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.096123882 +0000 UTC m=+79.282717166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:55.096205 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.096163    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:59.096155976 +0000 UTC m=+79.282749258 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:17:55.398768 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:55.398681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:55.398768 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:55.398752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:55.398990 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.398804    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:55.398990 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.398871    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.398853555 +0000 UTC m=+76.585446852 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:55.398990 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.398873    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:55.398990 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:55.398939    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:56.398925145 +0000 UTC m=+76.585518432 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:56.390004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:56.389970    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-647fs_8267b0b1-b622-489a-878d-6061b47da927/dns-node-resolver/0.log"
Jul 28 17:17:56.406403 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:56.406376    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:56.406513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:56.406436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:56.406513 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:56.406495    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:56.406631 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:56.406535    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:58.4065241 +0000 UTC m=+78.593117384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:56.406631 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:56.406537    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:56.406710 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:56.406649    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:58.406632693 +0000 UTC m=+78.593225976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:57.399520 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:57.399492    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5blq_b05ffd10-3b28-4ae6-9fd5-89b15f0f4139/node-ca/0.log"
Jul 28 17:17:57.793928 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:57.793900    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrwds_e434feb5-2d72-4980-a231-12e3ca1449c2/serve-healthcheck-canary/0.log"
Jul 28 17:17:58.422199 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:58.422159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:58.422667 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:58.422256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:17:58.422667 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:58.422289    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:58.422667 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:58.422357    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:02.422340819 +0000 UTC m=+82.608934123 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:58.422667 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:58.422376    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:58.422667 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:58.422428    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:02.422413252 +0000 UTC m=+82.609006551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:59.127198 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:59.127160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:59.127198 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:17:59.127197    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:17:59.127469 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:59.127349    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:17:59.127469 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:59.127350    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:07.127328789 +0000 UTC m=+87.313922072 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:17:59.127469 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:17:59.127402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:07.127387983 +0000 UTC m=+87.313981271 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:18:02.453588 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:02.453528    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:02.454019 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:02.453624    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:02.454019 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:02.453674    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:02.454019 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:02.453691    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:18:02.454019 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:02.453742    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:10.453726654 +0000 UTC m=+90.640319936 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:02.454019 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:02.453756    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:10.453749984 +0000 UTC m=+90.640343267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : configmap "kube-rbac-proxy" not found
Jul 28 17:18:07.196188 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:07.196155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:07.196188 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:07.196189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:07.196728 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:07.196320    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 28 17:18:07.196728 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:07.196361    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.196335126 +0000 UTC m=+103.382928417 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : configmap references non-existent config key: service-ca.crt
Jul 28 17:18:07.196728 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:07.196389    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs podName:0d3cc9dd-9a09-4852-8f44-50c87d93c575 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:23.196378677 +0000 UTC m=+103.382971959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs") pod "router-default-84996d74f-sgkml" (UID: "0d3cc9dd-9a09-4852-8f44-50c87d93c575") : secret "router-metrics-certs-default" not found
Jul 28 17:18:10.520773 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:10.520729    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:10.521259 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:10.520916    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:10.521259 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:10.521066    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:10.521259 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:10.521133    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls podName:465c3991-57e5-4a79-8a94-78763ce4180e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:26.521115852 +0000 UTC m=+106.707709139 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f7nmc" (UID: "465c3991-57e5-4a79-8a94-78763ce4180e") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:18:10.521259 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:10.521189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/465c3991-57e5-4a79-8a94-78763ce4180e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:18.582463 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.582427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:18:18.582961 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.582498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") pod \"image-registry-5cc55c59d-spwbb\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") " pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:18:18.582961 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:18.582599    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:18:18.582961 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:18.582615    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cc55c59d-spwbb: secret "image-registry-tls" not found
Jul 28 17:18:18.582961 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:18.582675    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls podName:7877be8c-9684-4285-a599-34cfa8cdd585 nodeName:}" failed. No retries permitted until 2026-07-28 17:19:22.582658375 +0000 UTC m=+162.769251657 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls") pod "image-registry-5cc55c59d-spwbb" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585") : secret "image-registry-tls" not found
Jul 28 17:18:18.584730 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.584708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/988ee349-bc90-4a64-9787-a4b7688e5f19-metrics-tls\") pod \"dns-default-jnqnd\" (UID: \"988ee349-bc90-4a64-9787-a4b7688e5f19\") " pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:18:18.613860 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.613833    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qjb4d\""
Jul 28 17:18:18.621294 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.621265    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:18:18.749606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:18.749575    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jnqnd"]
Jul 28 17:18:18.752867 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:18.752839    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod988ee349_bc90_4a64_9787_a4b7688e5f19.slice/crio-66570465b7d35da02cf02e1a783a5fa7e72c2140d78d91842a38c9d997b6007d WatchSource:0}: Error finding container 66570465b7d35da02cf02e1a783a5fa7e72c2140d78d91842a38c9d997b6007d: Status 404 returned error can't find the container with id 66570465b7d35da02cf02e1a783a5fa7e72c2140d78d91842a38c9d997b6007d
Jul 28 17:18:19.681089 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:19.681052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnqnd" event={"ID":"988ee349-bc90-4a64-9787-a4b7688e5f19","Type":"ContainerStarted","Data":"66570465b7d35da02cf02e1a783a5fa7e72c2140d78d91842a38c9d997b6007d"}
Jul 28 17:18:20.685237 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:20.685195    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnqnd" event={"ID":"988ee349-bc90-4a64-9787-a4b7688e5f19","Type":"ContainerStarted","Data":"3ab39ff4a98a8b16ce5bd85e4d1af6f54091d64a3725e99e16555e3942031962"}
Jul 28 17:18:20.685662 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:20.685251    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jnqnd" event={"ID":"988ee349-bc90-4a64-9787-a4b7688e5f19","Type":"ContainerStarted","Data":"f7ebfe1c2f47adbd3960f2af2aadd8ba0099482069dc353bd7e240a95b4b29bc"}
Jul 28 17:18:20.685662 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:20.685340    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:18:20.705990 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:20.705940    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jnqnd" podStartSLOduration=65.251148623 podStartE2EDuration="1m6.705924654s" totalImagesPullingTime="1.454776031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:20.7043144 +0000 UTC m=+100.890907706" watchObservedRunningTime="2026-07-28 17:18:20.705924654 +0000 UTC m=+100.892517959"
Jul 28 17:18:22.616627 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:22.616594    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-k5p9w"
Jul 28 17:18:23.218834 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.218800    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:23.218834 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.218835    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:23.219445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.219420    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d3cc9dd-9a09-4852-8f44-50c87d93c575-service-ca-bundle\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:23.221091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.221064    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0d3cc9dd-9a09-4852-8f44-50c87d93c575-metrics-certs\") pod \"router-default-84996d74f-sgkml\" (UID: \"0d3cc9dd-9a09-4852-8f44-50c87d93c575\") " pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:23.484471 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.484374    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:23.639193 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.639159    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-84996d74f-sgkml"]
Jul 28 17:18:23.642478 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:23.642453    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3cc9dd_9a09_4852_8f44_50c87d93c575.slice/crio-49d6e7636792b46265bc37ffef4b97039ec7cd3fe20035c0aca1b711f4e844bf WatchSource:0}: Error finding container 49d6e7636792b46265bc37ffef4b97039ec7cd3fe20035c0aca1b711f4e844bf: Status 404 returned error can't find the container with id 49d6e7636792b46265bc37ffef4b97039ec7cd3fe20035c0aca1b711f4e844bf
Jul 28 17:18:23.693033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:23.693009    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84996d74f-sgkml" event={"ID":"0d3cc9dd-9a09-4852-8f44-50c87d93c575","Type":"ContainerStarted","Data":"49d6e7636792b46265bc37ffef4b97039ec7cd3fe20035c0aca1b711f4e844bf"}
Jul 28 17:18:24.698153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:24.698109    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-84996d74f-sgkml" event={"ID":"0d3cc9dd-9a09-4852-8f44-50c87d93c575","Type":"ContainerStarted","Data":"2a9a29cf183281809ff54e650cefd6a2291af0b956865a4c96ea9545b4f8f770"}
Jul 28 17:18:24.720940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:24.720890    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-84996d74f-sgkml" podStartSLOduration=33.720876988 podStartE2EDuration="33.720876988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:24.720071638 +0000 UTC m=+104.906664946" watchObservedRunningTime="2026-07-28 17:18:24.720876988 +0000 UTC m=+104.907470292"
Jul 28 17:18:25.485416 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:25.485380    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:25.487957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:25.487936    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:25.701826 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:25.701793    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:25.702999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:25.702980    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-84996d74f-sgkml"
Jul 28 17:18:26.544393 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:26.544359    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:26.546597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:26.546567    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/465c3991-57e5-4a79-8a94-78763ce4180e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f7nmc\" (UID: \"465c3991-57e5-4a79-8a94-78763ce4180e\") " pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:26.838274 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:26.838184    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f7nmc"
Jul 28 17:18:26.961015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:26.960983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f7nmc"]
Jul 28 17:18:26.963748 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:26.963723    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465c3991_57e5_4a79_8a94_78763ce4180e.slice/crio-bc00ac2c47c32d5755a3e0565b731004596d1511f7c3cfb201965ad20164604b WatchSource:0}: Error finding container bc00ac2c47c32d5755a3e0565b731004596d1511f7c3cfb201965ad20164604b: Status 404 returned error can't find the container with id bc00ac2c47c32d5755a3e0565b731004596d1511f7c3cfb201965ad20164604b
Jul 28 17:18:27.710855 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:27.710768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7nmc" event={"ID":"465c3991-57e5-4a79-8a94-78763ce4180e","Type":"ContainerStarted","Data":"bc00ac2c47c32d5755a3e0565b731004596d1511f7c3cfb201965ad20164604b"}
Jul 28 17:18:28.940901 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.940864    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-448sj"]
Jul 28 17:18:28.943098 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.943075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:28.946923 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.946890    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 28 17:18:28.946923 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.946890    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-w2lzl\""
Jul 28 17:18:28.947109 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.946947    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 28 17:18:28.954533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:28.954509    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-448sj"]
Jul 28 17:18:29.066290 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.066263    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669afc54-8ae9-44f2-bf01-c6d3ba582a62-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.066429 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.066305    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/669afc54-8ae9-44f2-bf01-c6d3ba582a62-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.167071 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.167036    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669afc54-8ae9-44f2-bf01-c6d3ba582a62-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.167244 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.167077    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/669afc54-8ae9-44f2-bf01-c6d3ba582a62-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.167876 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.167846    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/669afc54-8ae9-44f2-bf01-c6d3ba582a62-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.169917 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.169890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/669afc54-8ae9-44f2-bf01-c6d3ba582a62-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-448sj\" (UID: \"669afc54-8ae9-44f2-bf01-c6d3ba582a62\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.253822 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.253788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj"
Jul 28 17:18:29.391895 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:29.391861    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-448sj"]
Jul 28 17:18:30.689851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:30.689822    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jnqnd"
Jul 28 17:18:31.234177 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:31.234143    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod669afc54_8ae9_44f2_bf01_c6d3ba582a62.slice/crio-fa19799a262757d49ba07290309be37b7227b6028203a4231c97834a9f209bed WatchSource:0}: Error finding container fa19799a262757d49ba07290309be37b7227b6028203a4231c97834a9f209bed: Status 404 returned error can't find the container with id fa19799a262757d49ba07290309be37b7227b6028203a4231c97834a9f209bed
Jul 28 17:18:31.724854 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:31.724814    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj" event={"ID":"669afc54-8ae9-44f2-bf01-c6d3ba582a62","Type":"ContainerStarted","Data":"fa19799a262757d49ba07290309be37b7227b6028203a4231c97834a9f209bed"}
Jul 28 17:18:32.734924 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:32.734881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7nmc" event={"ID":"465c3991-57e5-4a79-8a94-78763ce4180e","Type":"ContainerStarted","Data":"ffa4180297772aa602916517f25df034bca5cd58d0eb1c521c9fcc6f3b1c65cc"}
Jul 28 17:18:32.736700 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:32.736646    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj" event={"ID":"669afc54-8ae9-44f2-bf01-c6d3ba582a62","Type":"ContainerStarted","Data":"b1ed3225a6823c9f61e11eb5a60381a5e7a1e4741e9130225018587ccdc86990"}
Jul 28 17:18:35.748521 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.748441    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7nmc" event={"ID":"465c3991-57e5-4a79-8a94-78763ce4180e","Type":"ContainerStarted","Data":"746877ee1b63e9924d989f8b93b61e0f887092ce55d2701356db73dd91095f93"}
Jul 28 17:18:35.971342 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.971281    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-448sj" podStartSLOduration=6.677443251 podStartE2EDuration="7.971262157s" totalImagesPullingTime="1.293818906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:32.755309733 +0000 UTC m=+112.941903038" watchObservedRunningTime="2026-07-28 17:18:35.971262157 +0000 UTC m=+116.157855509"
Jul 28 17:18:35.971513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.971419    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xhljh"]
Jul 28 17:18:35.973626 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.973592    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:35.977397 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977367    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:35.977493 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977407    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:18:35.977493 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:18:35.977493 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977455    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:18:35.977493 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977479    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-9bfts\""
Jul 28 17:18:35.977783 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.977755    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:18:35.984942 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:35.984917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xhljh"]
Jul 28 17:18:36.019160 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.019118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491c4b66-2a14-4929-86fa-494c5b67e8f3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.019284 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.019166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.019284 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.019253    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf47g\" (UniqueName: \"kubernetes.io/projected/491c4b66-2a14-4929-86fa-494c5b67e8f3-kube-api-access-xf47g\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.019389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.019336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.120626 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.120585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491c4b66-2a14-4929-86fa-494c5b67e8f3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.120801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.120644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.120801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.120690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf47g\" (UniqueName: \"kubernetes.io/projected/491c4b66-2a14-4929-86fa-494c5b67e8f3-kube-api-access-xf47g\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.120801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.120724    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.121433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.121408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/491c4b66-2a14-4929-86fa-494c5b67e8f3-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.123390 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.123367    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.123488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.123441    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/491c4b66-2a14-4929-86fa-494c5b67e8f3-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.132511 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.132480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf47g\" (UniqueName: \"kubernetes.io/projected/491c4b66-2a14-4929-86fa-494c5b67e8f3-kube-api-access-xf47g\") pod \"prometheus-operator-84d69f794d-xhljh\" (UID: \"491c4b66-2a14-4929-86fa-494c5b67e8f3\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.285795 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.285708    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh"
Jul 28 17:18:36.434801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.434766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-xhljh"]
Jul 28 17:18:36.438226 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:36.438198    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491c4b66_2a14_4929_86fa_494c5b67e8f3.slice/crio-cdd320ac540cacbe5e475b116f13ad1d0d1e802a9df5e108e43d7250eb6b3c8e WatchSource:0}: Error finding container cdd320ac540cacbe5e475b116f13ad1d0d1e802a9df5e108e43d7250eb6b3c8e: Status 404 returned error can't find the container with id cdd320ac540cacbe5e475b116f13ad1d0d1e802a9df5e108e43d7250eb6b3c8e
Jul 28 17:18:36.752803 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:36.752764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh" event={"ID":"491c4b66-2a14-4929-86fa-494c5b67e8f3","Type":"ContainerStarted","Data":"cdd320ac540cacbe5e475b116f13ad1d0d1e802a9df5e108e43d7250eb6b3c8e"}
Jul 28 17:18:39.766173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:39.766141    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh" event={"ID":"491c4b66-2a14-4929-86fa-494c5b67e8f3","Type":"ContainerStarted","Data":"cb0be4c332b8b6fd0c60fe34afd45727027dab514f75f3f8d078c8fb3a12ec8c"}
Jul 28 17:18:39.766634 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:39.766228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh" event={"ID":"491c4b66-2a14-4929-86fa-494c5b67e8f3","Type":"ContainerStarted","Data":"96fdb072c4ddd07a8dd72a415f5d3e3f9887fe097713bac2b26178678a370094"}
Jul 28 17:18:39.768071 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:39.768047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f7nmc" event={"ID":"465c3991-57e5-4a79-8a94-78763ce4180e","Type":"ContainerStarted","Data":"9d0f9c6ef01140ba951dbcedf3295314df854b41e867a45bc985872b346a116e"}
Jul 28 17:18:39.784581 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:39.784500    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-xhljh" podStartSLOduration=1.97696253 podStartE2EDuration="4.784486315s" totalImagesPullingTime="2.807523785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:39.783630737 +0000 UTC m=+119.970224042" watchObservedRunningTime="2026-07-28 17:18:39.784486315 +0000 UTC m=+119.971079599"
Jul 28 17:18:39.802771 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:39.802730    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f7nmc" podStartSLOduration=33.617420847 podStartE2EDuration="45.802715923s" totalImagesPullingTime="12.185295076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:39.801630453 +0000 UTC m=+119.988223758" watchObservedRunningTime="2026-07-28 17:18:39.802715923 +0000 UTC m=+119.989309228"
Jul 28 17:18:41.389925 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.389894    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fwsp8"]
Jul 28 17:18:41.393293 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.393272    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.396630 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.396613    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:41.396731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.396612    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v97v9\""
Jul 28 17:18:41.397477 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.397459    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:41.401047 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.401025    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:41.404474 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.404454    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"]
Jul 28 17:18:41.407270 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.407254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.410205 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.410189    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 28 17:18:41.410872 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.410856    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 28 17:18:41.410872 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.410864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-ptb9g\""
Jul 28 17:18:41.414670 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.414648    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 28 17:18:41.421365 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.421346    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"]
Jul 28 17:18:41.462322 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-sys\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462322 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.462482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462363    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-wtmp\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.462482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr8t6\" (UniqueName: \"kubernetes.io/projected/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-api-access-hr8t6\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.462482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.462482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462474    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-textfile\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4edee1e3-e11c-4092-8aaf-6e350260ccd4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.462661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462538    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-root\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462811 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462666    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-metrics-client-ca\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462811 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb78v\" (UniqueName: \"kubernetes.io/projected/6b1c597a-6421-4dda-ba0b-105daa48681c-kube-api-access-wb78v\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.462811 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.462698    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.563763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563728    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.563763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr8t6\" (UniqueName: \"kubernetes.io/projected/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-api-access-hr8t6\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.563978 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563790    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.563978 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-textfile\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.563978 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563961    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4edee1e3-e11c-4092-8aaf-6e350260ccd4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-root\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.563998    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564031    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-metrics-client-ca\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb78v\" (UniqueName: \"kubernetes.io/projected/6b1c597a-6421-4dda-ba0b-105daa48681c-kube-api-access-wb78v\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564407 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564106    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-root\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564407 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564258    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-textfile\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564407 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.564407 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:41.564358    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:18:41.564407 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564383    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-sys\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:41.564420    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls podName:6b1c597a-6421-4dda-ba0b-105daa48681c nodeName:}" failed. No retries permitted until 2026-07-28 17:18:42.06439813 +0000 UTC m=+122.250991430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls") pod "node-exporter-fwsp8" (UID: "6b1c597a-6421-4dda-ba0b-105daa48681c") : secret "node-exporter-tls" not found
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-sys\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564439    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-wtmp\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564645    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.564698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-wtmp\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564997 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564693    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4edee1e3-e11c-4092-8aaf-6e350260ccd4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.564997 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564730    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-accelerators-collector-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.564997 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.564947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.565149 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.565116    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6b1c597a-6421-4dda-ba0b-105daa48681c-metrics-client-ca\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.566879 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.566843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.566996 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.566885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.566996 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.566973    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.578588 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.578566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb78v\" (UniqueName: \"kubernetes.io/projected/6b1c597a-6421-4dda-ba0b-105daa48681c-kube-api-access-wb78v\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:41.578868 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.578850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr8t6\" (UniqueName: \"kubernetes.io/projected/4edee1e3-e11c-4092-8aaf-6e350260ccd4-kube-api-access-hr8t6\") pod \"kube-state-metrics-56d9f944b4-t7j7j\" (UID: \"4edee1e3-e11c-4092-8aaf-6e350260ccd4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.715147 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.715053    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"
Jul 28 17:18:41.846333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:41.846301    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j"]
Jul 28 17:18:41.848974 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:41.848942    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edee1e3_e11c_4092_8aaf_6e350260ccd4.slice/crio-b47100c0aca9335bfce895732af213884dbdc88582c8425032480c784a70b9b4 WatchSource:0}: Error finding container b47100c0aca9335bfce895732af213884dbdc88582c8425032480c784a70b9b4: Status 404 returned error can't find the container with id b47100c0aca9335bfce895732af213884dbdc88582c8425032480c784a70b9b4
Jul 28 17:18:42.069351 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:42.069311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:42.071568 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:42.071531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6b1c597a-6421-4dda-ba0b-105daa48681c-node-exporter-tls\") pod \"node-exporter-fwsp8\" (UID: \"6b1c597a-6421-4dda-ba0b-105daa48681c\") " pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:42.301610 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:42.301578    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fwsp8"
Jul 28 17:18:42.312799 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:42.312769    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1c597a_6421_4dda_ba0b_105daa48681c.slice/crio-14ca17a4f4b931ab3c92ce6d2778ed5074e21f10c6a1a96c2b487dd84da287d4 WatchSource:0}: Error finding container 14ca17a4f4b931ab3c92ce6d2778ed5074e21f10c6a1a96c2b487dd84da287d4: Status 404 returned error can't find the container with id 14ca17a4f4b931ab3c92ce6d2778ed5074e21f10c6a1a96c2b487dd84da287d4
Jul 28 17:18:42.781084 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:42.781041    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwsp8" event={"ID":"6b1c597a-6421-4dda-ba0b-105daa48681c","Type":"ContainerStarted","Data":"14ca17a4f4b931ab3c92ce6d2778ed5074e21f10c6a1a96c2b487dd84da287d4"}
Jul 28 17:18:42.782528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:42.782492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j" event={"ID":"4edee1e3-e11c-4092-8aaf-6e350260ccd4","Type":"ContainerStarted","Data":"b47100c0aca9335bfce895732af213884dbdc88582c8425032480c784a70b9b4"}
Jul 28 17:18:43.787370 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.787339    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b1c597a-6421-4dda-ba0b-105daa48681c" containerID="a1cdc3c76a0b07176dca0c6ea47982aafce2e5e72e57c01d807befc95b4f0b34" exitCode=0
Jul 28 17:18:43.787869 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.787427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwsp8" event={"ID":"6b1c597a-6421-4dda-ba0b-105daa48681c","Type":"ContainerDied","Data":"a1cdc3c76a0b07176dca0c6ea47982aafce2e5e72e57c01d807befc95b4f0b34"}
Jul 28 17:18:43.789713 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.789686    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j" event={"ID":"4edee1e3-e11c-4092-8aaf-6e350260ccd4","Type":"ContainerStarted","Data":"2745e26f56814fde0f5985a5f11f83b657375fb4d8b42fa02e4a543968af2e95"}
Jul 28 17:18:43.789831 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.789743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j" event={"ID":"4edee1e3-e11c-4092-8aaf-6e350260ccd4","Type":"ContainerStarted","Data":"d101087435beab5090a41ef142a6bb6cc1d82bd242c614968c7e8ba89da7cf01"}
Jul 28 17:18:43.789831 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.789775    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j" event={"ID":"4edee1e3-e11c-4092-8aaf-6e350260ccd4","Type":"ContainerStarted","Data":"7bd3acdc4018b2f751fd9d253bdcb81ec206291d46afcb4c26ad1756babee2e3"}
Jul 28 17:18:43.827392 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:43.827345    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-t7j7j" podStartSLOduration=1.275042942 podStartE2EDuration="2.82732879s" totalImagesPullingTime="1.552285848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:43.826188412 +0000 UTC m=+124.012781716" watchObservedRunningTime="2026-07-28 17:18:43.82732879 +0000 UTC m=+124.013922095"
Jul 28 17:18:44.795434 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:44.795392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwsp8" event={"ID":"6b1c597a-6421-4dda-ba0b-105daa48681c","Type":"ContainerStarted","Data":"aa9f4f18df5877cf172afc3378a73120e02e056c8eea0e2afb1628054e68fb5e"}
Jul 28 17:18:44.795801 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:44.795460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fwsp8" event={"ID":"6b1c597a-6421-4dda-ba0b-105daa48681c","Type":"ContainerStarted","Data":"65887f8be801e823dada5d3242da59c48dd6026d6372d256893b78fba30f3e15"}
Jul 28 17:18:44.817749 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:44.817700    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fwsp8" podStartSLOduration=2.647210792 podStartE2EDuration="3.817686459s" totalImagesPullingTime="1.090608788s" totalInitContainerRuntime="79.866879ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:44.81635977 +0000 UTC m=+125.002953077" watchObservedRunningTime="2026-07-28 17:18:44.817686459 +0000 UTC m=+125.004279764"
Jul 28 17:18:46.599954 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.599923    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"]
Jul 28 17:18:46.603628 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.603604    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.607112 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.607089    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-8n7lt\""
Jul 28 17:18:46.607222 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.607174    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:18:46.607628 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.607610    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:18:46.608165 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.608143    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:18:46.608258 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.608183    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:18:46.608258 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.608155    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:18:46.613170 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.613151    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:18:46.614314 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.614297    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"]
Jul 28 17:18:46.707869 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.707837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-federate-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708051 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.707876    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708051 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.707918    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708051 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.707966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-metrics-client-ca\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708051 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.708026    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708244 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.708075    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-serving-certs-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708244 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.708113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.708244 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.708147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9ct\" (UniqueName: \"kubernetes.io/projected/001f3fa4-e221-4daf-8e6d-c67064648f88-kube-api-access-5h9ct\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808498 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9ct\" (UniqueName: \"kubernetes.io/projected/001f3fa4-e221-4daf-8e6d-c67064648f88-kube-api-access-5h9ct\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808519    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-federate-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-metrics-client-ca\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808681    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808934 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-serving-certs-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.808934 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.808752    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.809418 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.809390    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.809601 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.809435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-metrics-client-ca\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.809601 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.809434    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/001f3fa4-e221-4daf-8e6d-c67064648f88-serving-certs-ca-bundle\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.811238 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.811210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-federate-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.811238 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.811232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.811445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.811425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-telemeter-client-tls\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.811646 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.811624    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/001f3fa4-e221-4daf-8e6d-c67064648f88-secret-telemeter-client\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.820481 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.820462    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9ct\" (UniqueName: \"kubernetes.io/projected/001f3fa4-e221-4daf-8e6d-c67064648f88-kube-api-access-5h9ct\") pod \"telemeter-client-5b8fcb596-cdsg8\" (UID: \"001f3fa4-e221-4daf-8e6d-c67064648f88\") " pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:46.914390 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:46.914312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"
Jul 28 17:18:47.047652 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:47.047627    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5b8fcb596-cdsg8"]
Jul 28 17:18:47.050022 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:47.049991    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001f3fa4_e221_4daf_8e6d_c67064648f88.slice/crio-d187b9ce5a3b4bcb0d31528b9b939294d6466105de2555e6ef37084f520a684c WatchSource:0}: Error finding container d187b9ce5a3b4bcb0d31528b9b939294d6466105de2555e6ef37084f520a684c: Status 404 returned error can't find the container with id d187b9ce5a3b4bcb0d31528b9b939294d6466105de2555e6ef37084f520a684c
Jul 28 17:18:47.809437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:47.809395    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8" event={"ID":"001f3fa4-e221-4daf-8e6d-c67064648f88","Type":"ContainerStarted","Data":"d187b9ce5a3b4bcb0d31528b9b939294d6466105de2555e6ef37084f520a684c"}
Jul 28 17:18:48.637446 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.637417    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:18:48.640976 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.640955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.643672 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.643651    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:18:48.644759 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.644645    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:18:48.644759 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.644656    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66zv9\""
Jul 28 17:18:48.644759 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.644657    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:18:48.645518 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.645493    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:18:48.646440 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.646056    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:18:48.647346 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.646660    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:18:48.647346 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.647150    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:18:48.654060 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.654032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:18:48.656314 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.656261    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:18:48.727212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727177    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727306    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvbbl\" (UniqueName: \"kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727362    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727576 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.727576 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.727446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828491 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828518    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvbbl\" (UniqueName: \"kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828670    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828718    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.828967 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.828766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.829303 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.829279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.829463 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.829438    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.829579 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.829478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.829579 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.829478    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.831684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.831661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.831783 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.831689    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.838583 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.838535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvbbl\" (UniqueName: \"kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl\") pod \"console-868c694d85-h4thn\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") " pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:48.954430 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:48.954356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:49.208089 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:49.208017    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:18:49.210887 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:49.210861    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod141a95e1_b0f2_4b53_aae3_6e589929f702.slice/crio-2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252 WatchSource:0}: Error finding container 2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252: Status 404 returned error can't find the container with id 2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252
Jul 28 17:18:49.819700 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:49.819651    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8" event={"ID":"001f3fa4-e221-4daf-8e6d-c67064648f88","Type":"ContainerStarted","Data":"228baaa8ec0f8c5de80de2a4a7310abe8a4039e47b19a321f15a75221f12390e"}
Jul 28 17:18:49.821092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:49.821068    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c694d85-h4thn" event={"ID":"141a95e1-b0f2-4b53-aae3-6e589929f702","Type":"ContainerStarted","Data":"2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252"}
Jul 28 17:18:50.040318 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.040278    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:18:50.045619 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.043829    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5333fe-432d-40d7-a893-35a95b16181f-metrics-certs\") pod \"network-metrics-daemon-tqq8n\" (UID: \"4b5333fe-432d-40d7-a893-35a95b16181f\") " pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:18:50.174041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.173959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vvwzr\""
Jul 28 17:18:50.181754 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.181573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqq8n"
Jul 28 17:18:50.614160 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.614128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqq8n"]
Jul 28 17:18:50.617350 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:18:50.617319    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5333fe_432d_40d7_a893_35a95b16181f.slice/crio-c09250290838c52e4b998c2ac368655e650d2453e1e4998727f9f43a37e23eb9 WatchSource:0}: Error finding container c09250290838c52e4b998c2ac368655e650d2453e1e4998727f9f43a37e23eb9: Status 404 returned error can't find the container with id c09250290838c52e4b998c2ac368655e650d2453e1e4998727f9f43a37e23eb9
Jul 28 17:18:50.828128 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.828089    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8" event={"ID":"001f3fa4-e221-4daf-8e6d-c67064648f88","Type":"ContainerStarted","Data":"edb5f042b46e1978180c140224e9a237b4901aadc408886e3923743dde83d90c"}
Jul 28 17:18:50.828371 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.828167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8" event={"ID":"001f3fa4-e221-4daf-8e6d-c67064648f88","Type":"ContainerStarted","Data":"4be6cdb0e7a7ea63e64d02e6d2ace2115d60999d73ac6e9d0e74d390451d16bf"}
Jul 28 17:18:50.829809 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.829750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqq8n" event={"ID":"4b5333fe-432d-40d7-a893-35a95b16181f","Type":"ContainerStarted","Data":"c09250290838c52e4b998c2ac368655e650d2453e1e4998727f9f43a37e23eb9"}
Jul 28 17:18:50.857364 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:50.857301    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5b8fcb596-cdsg8" podStartSLOduration=1.450990688 podStartE2EDuration="4.857281111s" totalImagesPullingTime="3.406290423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:50.855532079 +0000 UTC m=+131.042125387" watchObservedRunningTime="2026-07-28 17:18:50.857281111 +0000 UTC m=+131.043874417"
Jul 28 17:18:52.839258 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:52.839152    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqq8n" event={"ID":"4b5333fe-432d-40d7-a893-35a95b16181f","Type":"ContainerStarted","Data":"04f14a0fb2f9c78fc52d89d998fd8d5932357733c6a532cf5eaaf2a4484f3e91"}
Jul 28 17:18:52.839258 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:52.839228    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqq8n" event={"ID":"4b5333fe-432d-40d7-a893-35a95b16181f","Type":"ContainerStarted","Data":"30c348c98e5851cb68df60a8cba8e2c7394e98871906511b939809fabbe6c5e7"}
Jul 28 17:18:52.840709 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:52.840681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c694d85-h4thn" event={"ID":"141a95e1-b0f2-4b53-aae3-6e589929f702","Type":"ContainerStarted","Data":"f5c8783b3185f8e7db0879d09ee48cd695ceb3f11106aa2a291c007555d30e08"}
Jul 28 17:18:52.865435 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:52.865388    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tqq8n" podStartSLOduration=130.950602061 podStartE2EDuration="2m12.865375601s" totalImagesPullingTime="1.91477354s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:52.863672706 +0000 UTC m=+133.050266012" watchObservedRunningTime="2026-07-28 17:18:52.865375601 +0000 UTC m=+133.051968906"
Jul 28 17:18:52.887275 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:52.887220    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-868c694d85-h4thn" podStartSLOduration=1.558731072 podStartE2EDuration="4.887203875s" totalImagesPullingTime="3.328472803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:52.885914423 +0000 UTC m=+133.072507728" watchObservedRunningTime="2026-07-28 17:18:52.887203875 +0000 UTC m=+133.073797180"
Jul 28 17:18:54.569873 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.569838    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cc55c59d-spwbb"]
Jul 28 17:18:54.570225 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:18:54.570068    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-5cc55c59d-spwbb" podUID="7877be8c-9684-4285-a599-34cfa8cdd585"
Jul 28 17:18:54.847379 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.847302    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:18:54.858157 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.858128    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:18:54.988151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988116    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tt48\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988234    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988265    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988281    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988301    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted\") pod \"7877be8c-9684-4285-a599-34cfa8cdd585\" (UID: \"7877be8c-9684-4285-a599-34cfa8cdd585\") "
Jul 28 17:18:54.988699 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988668    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:54.988699 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988682    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:18:54.988828 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.988717    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:54.990471 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.990440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48" (OuterVolumeSpecName: "kube-api-access-9tt48") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "kube-api-access-9tt48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:54.990624 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.990555    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:54.990775 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.990752    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:54.990878 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:54.990818    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "7877be8c-9684-4285-a599-34cfa8cdd585" (UID: "7877be8c-9684-4285-a599-34cfa8cdd585"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:55.088959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088921    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-registry-certificates\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.088959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088952    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-bound-sa-token\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.088959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088961    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7877be8c-9684-4285-a599-34cfa8cdd585-trusted-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.088959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088969    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/7877be8c-9684-4285-a599-34cfa8cdd585-ca-trust-extracted\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.089212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088977    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tt48\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-kube-api-access-9tt48\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.089212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088987    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-image-registry-private-configuration\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.089212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.088997    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/7877be8c-9684-4285-a599-34cfa8cdd585-installation-pull-secrets\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:55.850657 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.850627    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cc55c59d-spwbb"
Jul 28 17:18:55.885525 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.885492    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cc55c59d-spwbb"]
Jul 28 17:18:55.890148 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.890120    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cc55c59d-spwbb"]
Jul 28 17:18:55.995606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:55.995575    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/7877be8c-9684-4285-a599-34cfa8cdd585-registry-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:18:56.357115 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:56.357089    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7877be8c-9684-4285-a599-34cfa8cdd585" path="/var/lib/kubelet/pods/7877be8c-9684-4285-a599-34cfa8cdd585/volumes"
Jul 28 17:18:58.955387 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:58.955351    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:58.955798 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:58.955445    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:58.959909 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:58.959888    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:18:59.868585 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:18:59.868559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:19:02.713936 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:02.713901    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:19:27.732955 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.732896    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-868c694d85-h4thn" podUID="141a95e1-b0f2-4b53-aae3-6e589929f702" containerName="console" containerID="cri-o://f5c8783b3185f8e7db0879d09ee48cd695ceb3f11106aa2a291c007555d30e08" gracePeriod=15
Jul 28 17:19:27.965395 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.965366    2576 generic.go:356] "Generic (PLEG): container finished" podID="141a95e1-b0f2-4b53-aae3-6e589929f702" containerID="f5c8783b3185f8e7db0879d09ee48cd695ceb3f11106aa2a291c007555d30e08" exitCode=0
Jul 28 17:19:27.965507 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.965404    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c694d85-h4thn" event={"ID":"141a95e1-b0f2-4b53-aae3-6e589929f702","Type":"ContainerDied","Data":"f5c8783b3185f8e7db0879d09ee48cd695ceb3f11106aa2a291c007555d30e08"}
Jul 28 17:19:27.965507 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.965445    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-868c694d85-h4thn" event={"ID":"141a95e1-b0f2-4b53-aae3-6e589929f702","Type":"ContainerDied","Data":"2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252"}
Jul 28 17:19:27.965507 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.965461    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2231c67a176a2de86032ce9f91d835447f18d6b712eb38ccce86f77a2ea1d252"
Jul 28 17:19:27.970724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:27.970703    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:19:28.154306 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154272    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154306 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154312    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154592 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154347    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154592 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154405    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154592 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154592 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154502    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154592 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154575    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kvbbl\" (UniqueName: \"kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl\") pod \"141a95e1-b0f2-4b53-aae3-6e589929f702\" (UID: \"141a95e1-b0f2-4b53-aae3-6e589929f702\") "
Jul 28 17:19:28.154895 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154780    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config" (OuterVolumeSpecName: "console-config") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:28.154959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:28.155065 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.154942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca" (OuterVolumeSpecName: "service-ca") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:28.155180 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.155163    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:28.156869 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.156842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:28.156970 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.156890    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:28.156970 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.156898    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl" (OuterVolumeSpecName: "kube-api-access-kvbbl") pod "141a95e1-b0f2-4b53-aae3-6e589929f702" (UID: "141a95e1-b0f2-4b53-aae3-6e589929f702"). InnerVolumeSpecName "kube-api-access-kvbbl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:28.255349 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255312    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-console-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255349 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255343    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255349 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255352    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-oauth-serving-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255361    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/141a95e1-b0f2-4b53-aae3-6e589929f702-trusted-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255370    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-serving-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255378    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/141a95e1-b0f2-4b53-aae3-6e589929f702-console-oauth-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.255595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.255386    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvbbl\" (UniqueName: \"kubernetes.io/projected/141a95e1-b0f2-4b53-aae3-6e589929f702-kube-api-access-kvbbl\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:19:28.969168 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.969130    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-868c694d85-h4thn"
Jul 28 17:19:28.988138 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.988108    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:19:28.992655 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:28.992632    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-868c694d85-h4thn"]
Jul 28 17:19:30.356940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:19:30.356900    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="141a95e1-b0f2-4b53-aae3-6e589929f702" path="/var/lib/kubelet/pods/141a95e1-b0f2-4b53-aae3-6e589929f702/volumes"
Jul 28 17:20:24.303307 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.303268    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:20:24.303763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.303524    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="141a95e1-b0f2-4b53-aae3-6e589929f702" containerName="console"
Jul 28 17:20:24.303763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.303535    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="141a95e1-b0f2-4b53-aae3-6e589929f702" containerName="console"
Jul 28 17:20:24.303763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.303609    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="141a95e1-b0f2-4b53-aae3-6e589929f702" containerName="console"
Jul 28 17:20:24.309559 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.309523    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.312330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.312300    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:20:24.312437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.312339    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:20:24.312437 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.312413    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 28 17:20:24.313574 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.313535    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-66zv9\""
Jul 28 17:20:24.313684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.313579    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 28 17:20:24.313684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.313616    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 28 17:20:24.313684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.313677    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 28 17:20:24.313835 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.313677    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 28 17:20:24.318118 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.318095    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:20:24.318428 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.318408    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 28 17:20:24.453390 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453356    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5wzl\" (UniqueName: \"kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453399    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453498    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453569    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453604    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.453796 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.453649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.554752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554667    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.554752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.554752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554732    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5wzl\" (UniqueName: \"kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.554991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555608 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.555580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555788 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.555630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555788 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.555652    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.555907 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.555729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.557304 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.557275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.557381 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.557364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.563761 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.563740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5wzl\" (UniqueName: \"kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl\") pod \"console-6b994f76fc-bhznc\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") " pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.619774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.619735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:24.746430 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:24.746397    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:20:24.749299 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:20:24.749272    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49653f39_6de3_4f3f_a5fe_14c43017482d.slice/crio-a1ef9162fc251259a78ec861ab26ae2d5c430b2900e5ec741652975a220d3741 WatchSource:0}: Error finding container a1ef9162fc251259a78ec861ab26ae2d5c430b2900e5ec741652975a220d3741: Status 404 returned error can't find the container with id a1ef9162fc251259a78ec861ab26ae2d5c430b2900e5ec741652975a220d3741
Jul 28 17:20:25.166233 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:25.166148    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b994f76fc-bhznc" event={"ID":"49653f39-6de3-4f3f-a5fe-14c43017482d","Type":"ContainerStarted","Data":"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"}
Jul 28 17:20:25.166233 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:25.166198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b994f76fc-bhznc" event={"ID":"49653f39-6de3-4f3f-a5fe-14c43017482d","Type":"ContainerStarted","Data":"a1ef9162fc251259a78ec861ab26ae2d5c430b2900e5ec741652975a220d3741"}
Jul 28 17:20:25.185920 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:25.185867    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6b994f76fc-bhznc" podStartSLOduration=1.185850699 podStartE2EDuration="1.185850699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:20:25.184481974 +0000 UTC m=+225.371075274" watchObservedRunningTime="2026-07-28 17:20:25.185850699 +0000 UTC m=+225.372444003"
Jul 28 17:20:34.620473 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:34.620429    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:34.620972 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:34.620501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:34.625168 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:34.625147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:20:35.205112 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:20:35.205077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:21:02.139602 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.139561    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"]
Jul 28 17:21:02.142653 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.142633    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.145362 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.145334    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 17:21:02.145478 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.145361    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 17:21:02.146521 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.146503    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 17:21:02.146877 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.146859    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 17:21:02.146966 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.146880    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 17:21:02.146966 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.146892    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 17:21:02.146966 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.146896    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 17:21:02.152361 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.152339    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"]
Jul 28 17:21:02.230611 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/110cb331-9fd9-45d8-a4b0-37d9f5860528-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.230611 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230613    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zfcx\" (UniqueName: \"kubernetes.io/projected/110cb331-9fd9-45d8-a4b0-37d9f5860528-kube-api-access-6zfcx\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.230864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230635    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.230864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230662    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.230864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230742    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.230864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.230782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.331845 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/110cb331-9fd9-45d8-a4b0-37d9f5860528-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.331845 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zfcx\" (UniqueName: \"kubernetes.io/projected/110cb331-9fd9-45d8-a4b0-37d9f5860528-kube-api-access-6zfcx\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.332083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.332083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331904    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.332083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.332083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.331958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.332641 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.332601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/110cb331-9fd9-45d8-a4b0-37d9f5860528-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.334570 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.334531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.334570 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.334557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.334704 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.334629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-ca\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.334704 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.334665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/110cb331-9fd9-45d8-a4b0-37d9f5860528-hub\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.340919 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.340894    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zfcx\" (UniqueName: \"kubernetes.io/projected/110cb331-9fd9-45d8-a4b0-37d9f5860528-kube-api-access-6zfcx\") pod \"cluster-proxy-proxy-agent-56c884d798-lwccw\" (UID: \"110cb331-9fd9-45d8-a4b0-37d9f5860528\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.461089 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.460995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"
Jul 28 17:21:02.591042 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:02.591005    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw"]
Jul 28 17:21:02.594157 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:21:02.594130    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod110cb331_9fd9_45d8_a4b0_37d9f5860528.slice/crio-c1cb27ab1128a6b597a57802ea50a7a1ab6ec3aec88007a3978f695b47c57a6f WatchSource:0}: Error finding container c1cb27ab1128a6b597a57802ea50a7a1ab6ec3aec88007a3978f695b47c57a6f: Status 404 returned error can't find the container with id c1cb27ab1128a6b597a57802ea50a7a1ab6ec3aec88007a3978f695b47c57a6f
Jul 28 17:21:03.297668 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:03.297626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw" event={"ID":"110cb331-9fd9-45d8-a4b0-37d9f5860528","Type":"ContainerStarted","Data":"c1cb27ab1128a6b597a57802ea50a7a1ab6ec3aec88007a3978f695b47c57a6f"}
Jul 28 17:21:07.317010 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:07.316970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw" event={"ID":"110cb331-9fd9-45d8-a4b0-37d9f5860528","Type":"ContainerStarted","Data":"43f3afd1c6a40d7e65f78ffa0cfb11457d1e933308f56a90b03e138f6f85233f"}
Jul 28 17:21:07.317389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:07.317021    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw" event={"ID":"110cb331-9fd9-45d8-a4b0-37d9f5860528","Type":"ContainerStarted","Data":"ad2885b5db20072f9acd1ca2cc602d967d539140fa89b333298ecf2497a4a885"}
Jul 28 17:21:07.317389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:07.317042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw" event={"ID":"110cb331-9fd9-45d8-a4b0-37d9f5860528","Type":"ContainerStarted","Data":"41dbf464d08f3d14088484763cc0d53867bac568c01e8b8f3b71b391f5d818c8"}
Jul 28 17:21:07.339199 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:07.339144    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56c884d798-lwccw" podStartSLOduration=1.348689827 podStartE2EDuration="5.339127318s" totalImagesPullingTime="3.990437491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:07.338399319 +0000 UTC m=+267.524992630" watchObservedRunningTime="2026-07-28 17:21:07.339127318 +0000 UTC m=+267.525720623"
Jul 28 17:21:40.240727 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:40.240700    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:21:41.359763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.359727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-j82bp"]
Jul 28 17:21:41.361854 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.361837    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.364535 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.364511    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 28 17:21:41.364655 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.364561    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 28 17:21:41.364655 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.364556    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-lcrgr\""
Jul 28 17:21:41.365621 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.365602    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 28 17:21:41.365719 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.365620    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 28 17:21:41.365719 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.365604    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 28 17:21:41.371930 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.371907    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-j82bp"]
Jul 28 17:21:41.425916 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.425874    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.425916 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.425914    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d08a72ae-7fe5-4719-8e92-615a32ddd265-cabundle0\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.426121 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.426001    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87t5\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-kube-api-access-w87t5\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.526650 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.526611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.526650 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.526648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d08a72ae-7fe5-4719-8e92-615a32ddd265-cabundle0\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.526861 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.526691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w87t5\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-kube-api-access-w87t5\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.526861 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.526765    2576 projected.go:266] Couldn't get secret openshift-keda/keda-operator-certs: secret "keda-operator-certs" not found
Jul 28 17:21:41.526861 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.526791    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 28 17:21:41.526861 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.526799    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 17:21:41.526861 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.526811    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-j82bp: [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 17:21:41.527020 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.526866    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates podName:d08a72ae-7fe5-4719-8e92-615a32ddd265 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:42.026847004 +0000 UTC m=+302.213440299 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates") pod "keda-operator-77f648d694-j82bp" (UID: "d08a72ae-7fe5-4719-8e92-615a32ddd265") : [secret "keda-operator-certs" not found, references non-existent secret key: ca.crt]
Jul 28 17:21:41.527341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.527321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/d08a72ae-7fe5-4719-8e92-615a32ddd265-cabundle0\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.537481 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.537451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87t5\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-kube-api-access-w87t5\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:41.767432 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.767397    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"]
Jul 28 17:21:41.769486 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.769471    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.771954 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.771933    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 28 17:21:41.787298 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.787266    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"]
Jul 28 17:21:41.829357 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.829311    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghjhz\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-kube-api-access-ghjhz\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.829524 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.829439    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.829524 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.829501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.930903 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.930861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.931100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.930920    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghjhz\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-kube-api-access-ghjhz\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.931100 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.930976    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.931100 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.931037    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 28 17:21:41.931100 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.931061    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 17:21:41.931100 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.931082    2576 projected.go:266] Couldn't get secret openshift-keda/keda-metrics-apiserver-certs: secret "keda-metrics-apiserver-certs" not found
Jul 28 17:21:41.931343 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.931103    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p: [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 28 17:21:41.931343 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:41.931172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates podName:39cb77d5-a4a5-45d9-9f27-f36552f3c2ec nodeName:}" failed. No retries permitted until 2026-07-28 17:21:42.431151431 +0000 UTC m=+302.617744729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates") pod "keda-metrics-apiserver-55557df8c-sxm9p" (UID: "39cb77d5-a4a5-45d9-9f27-f36552f3c2ec") : [references non-existent secret key: tls.crt, secret "keda-metrics-apiserver-certs" not found]
Jul 28 17:21:41.931343 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.931322    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:41.942805 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:41.942774    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghjhz\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-kube-api-access-ghjhz\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:42.032341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:42.032247    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:42.032486 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.032377    2576 secret.go:281] references non-existent secret key: ca.crt
Jul 28 17:21:42.032486 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.032391    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 28 17:21:42.032486 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.032399    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-j82bp: references non-existent secret key: ca.crt
Jul 28 17:21:42.032486 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.032456    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates podName:d08a72ae-7fe5-4719-8e92-615a32ddd265 nodeName:}" failed. No retries permitted until 2026-07-28 17:21:43.032443435 +0000 UTC m=+303.219036717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates") pod "keda-operator-77f648d694-j82bp" (UID: "d08a72ae-7fe5-4719-8e92-615a32ddd265") : references non-existent secret key: ca.crt
Jul 28 17:21:42.435912 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:42.435840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:42.436274 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.435983    2576 secret.go:281] references non-existent secret key: tls.crt
Jul 28 17:21:42.436274 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.436001    2576 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 28 17:21:42.436274 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.436020    2576 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p: references non-existent secret key: tls.crt
Jul 28 17:21:42.436274 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:21:42.436081    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates podName:39cb77d5-a4a5-45d9-9f27-f36552f3c2ec nodeName:}" failed. No retries permitted until 2026-07-28 17:21:43.436066346 +0000 UTC m=+303.622659634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates") pod "keda-metrics-apiserver-55557df8c-sxm9p" (UID: "39cb77d5-a4a5-45d9-9f27-f36552f3c2ec") : references non-existent secret key: tls.crt
Jul 28 17:21:43.040499 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.040461    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:43.042915 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.042885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/d08a72ae-7fe5-4719-8e92-615a32ddd265-certificates\") pod \"keda-operator-77f648d694-j82bp\" (UID: \"d08a72ae-7fe5-4719-8e92-615a32ddd265\") " pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:43.172115 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.172075    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:43.314159 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.314081    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-j82bp"]
Jul 28 17:21:43.316957 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:21:43.316928    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08a72ae_7fe5_4719_8e92_615a32ddd265.slice/crio-a3c40098d68faf67c71dcbcf0622254edf9a1f9b0ec0a1645f6534b8c19e76c4 WatchSource:0}: Error finding container a3c40098d68faf67c71dcbcf0622254edf9a1f9b0ec0a1645f6534b8c19e76c4: Status 404 returned error can't find the container with id a3c40098d68faf67c71dcbcf0622254edf9a1f9b0ec0a1645f6534b8c19e76c4
Jul 28 17:21:43.318329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.318307    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:21:43.439642 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.439607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-j82bp" event={"ID":"d08a72ae-7fe5-4719-8e92-615a32ddd265","Type":"ContainerStarted","Data":"a3c40098d68faf67c71dcbcf0622254edf9a1f9b0ec0a1645f6534b8c19e76c4"}
Jul 28 17:21:43.444911 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.444886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:43.447316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.447299    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/39cb77d5-a4a5-45d9-9f27-f36552f3c2ec-certificates\") pod \"keda-metrics-apiserver-55557df8c-sxm9p\" (UID: \"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:43.579948 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.579854    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:43.704348 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:43.704323    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"]
Jul 28 17:21:43.706716 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:21:43.706679    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cb77d5_a4a5_45d9_9f27_f36552f3c2ec.slice/crio-3100c7e5cc8255782e94c23d3fe2c63e2ff295eefce6125132dcf1b5cd5d5450 WatchSource:0}: Error finding container 3100c7e5cc8255782e94c23d3fe2c63e2ff295eefce6125132dcf1b5cd5d5450: Status 404 returned error can't find the container with id 3100c7e5cc8255782e94c23d3fe2c63e2ff295eefce6125132dcf1b5cd5d5450
Jul 28 17:21:44.448041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:44.447924    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p" event={"ID":"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec","Type":"ContainerStarted","Data":"3100c7e5cc8255782e94c23d3fe2c63e2ff295eefce6125132dcf1b5cd5d5450"}
Jul 28 17:21:48.466377 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.466343    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p" event={"ID":"39cb77d5-a4a5-45d9-9f27-f36552f3c2ec","Type":"ContainerStarted","Data":"1a2b7d7c92bfdf27cc09d27334d787b63c7a40baa9d81933903b139b27900dd7"}
Jul 28 17:21:48.466866 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.466469    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:21:48.467867 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.467842    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-j82bp" event={"ID":"d08a72ae-7fe5-4719-8e92-615a32ddd265","Type":"ContainerStarted","Data":"517a26b636ab059ec7fcc67d5840afdebe4efbb7c2f167466396300ee5ea6f13"}
Jul 28 17:21:48.467978 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.467895    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:21:48.487315 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.487273    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p" podStartSLOduration=3.563952202 podStartE2EDuration="7.487261238s" totalImagesPullingTime="3.923309036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:48.486370061 +0000 UTC m=+308.672963367" watchObservedRunningTime="2026-07-28 17:21:48.487261238 +0000 UTC m=+308.673854543"
Jul 28 17:21:48.512330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:48.512282    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-j82bp" podStartSLOduration=3.199167722 podStartE2EDuration="7.512261258s" totalImagesPullingTime="4.313093536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:21:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:21:48.512018339 +0000 UTC m=+308.698611643" watchObservedRunningTime="2026-07-28 17:21:48.512261258 +0000 UTC m=+308.698854575"
Jul 28 17:21:59.476187 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:21:59.476157    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-sxm9p"
Jul 28 17:22:09.474388 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:09.474359    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-j82bp"
Jul 28 17:22:57.774804 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.774728    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-fv9lk"]
Jul 28 17:22:57.776887 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.776871    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:57.779584 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.779534    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:22:57.780691 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.780671    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:22:57.780784 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.780671    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-j2xjc\""
Jul 28 17:22:57.780784 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.780671    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:22:57.788326 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.788303    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fv9lk"]
Jul 28 17:22:57.923555 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.923517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shtrm\" (UniqueName: \"kubernetes.io/projected/03dae415-ddff-4cc6-8000-7db321a248aa-kube-api-access-shtrm\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:57.923737 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:57.923672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03dae415-ddff-4cc6-8000-7db321a248aa-data\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.024915 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.024814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03dae415-ddff-4cc6-8000-7db321a248aa-data\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.024915 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.024874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shtrm\" (UniqueName: \"kubernetes.io/projected/03dae415-ddff-4cc6-8000-7db321a248aa-kube-api-access-shtrm\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.025224 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.025205    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/03dae415-ddff-4cc6-8000-7db321a248aa-data\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.034525 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.034493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shtrm\" (UniqueName: \"kubernetes.io/projected/03dae415-ddff-4cc6-8000-7db321a248aa-kube-api-access-shtrm\") pod \"seaweedfs-64568bcd49-fv9lk\" (UID: \"03dae415-ddff-4cc6-8000-7db321a248aa\") " pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.086529 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.086484    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:22:58.213815 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.213781    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fv9lk"]
Jul 28 17:22:58.216882 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:22:58.216845    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03dae415_ddff_4cc6_8000_7db321a248aa.slice/crio-ba1b781b46330d4a6e93dc942473ae9706f0d94752779e942a1f529722b9ce5a WatchSource:0}: Error finding container ba1b781b46330d4a6e93dc942473ae9706f0d94752779e942a1f529722b9ce5a: Status 404 returned error can't find the container with id ba1b781b46330d4a6e93dc942473ae9706f0d94752779e942a1f529722b9ce5a
Jul 28 17:22:58.716603 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:22:58.716570    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fv9lk" event={"ID":"03dae415-ddff-4cc6-8000-7db321a248aa","Type":"ContainerStarted","Data":"ba1b781b46330d4a6e93dc942473ae9706f0d94752779e942a1f529722b9ce5a"}
Jul 28 17:23:01.731219 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:23:01.731132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fv9lk" event={"ID":"03dae415-ddff-4cc6-8000-7db321a248aa","Type":"ContainerStarted","Data":"78ff5ce177bddbc263af55f1a0b05da96084c9e5959e25c03ae92b64bbd4ab8f"}
Jul 28 17:23:01.731613 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:23:01.731254    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:23:01.752027 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:23:01.751974    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-fv9lk" podStartSLOduration=1.576030341 podStartE2EDuration="4.7519604s" totalImagesPullingTime="3.175930059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:01.75056898 +0000 UTC m=+381.937162279" watchObservedRunningTime="2026-07-28 17:23:01.7519604 +0000 UTC m=+381.938553705"
Jul 28 17:23:07.738458 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:23:07.738427    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-fv9lk"
Jul 28 17:24:10.167374 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:10.167282    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:24:13.461278 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.461239    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7k9s2"]
Jul 28 17:24:13.464325 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.464304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.467000 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.466965    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 17:24:13.467103 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.467042    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-76g8k\""
Jul 28 17:24:13.474637 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.474612    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7k9s2"]
Jul 28 17:24:13.626104 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.626065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90ba8e68-8771-4d00-bba5-96d640379eb7-cert\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.626275 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.626119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/90ba8e68-8771-4d00-bba5-96d640379eb7-kube-api-access-x85rv\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.727199 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.727106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90ba8e68-8771-4d00-bba5-96d640379eb7-cert\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.727344 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.727211    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/90ba8e68-8771-4d00-bba5-96d640379eb7-kube-api-access-x85rv\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.730375 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.730347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/90ba8e68-8771-4d00-bba5-96d640379eb7-cert\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.738338 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.738317    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x85rv\" (UniqueName: \"kubernetes.io/projected/90ba8e68-8771-4d00-bba5-96d640379eb7-kube-api-access-x85rv\") pod \"odh-model-controller-75d9cdbddf-7k9s2\" (UID: \"90ba8e68-8771-4d00-bba5-96d640379eb7\") " pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.774943 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.774908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:13.901012 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.900980    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-7k9s2"]
Jul 28 17:24:13.903332 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:24:13.903303    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ba8e68_8771_4d00_bba5_96d640379eb7.slice/crio-424cd40d6e1937f0ec144b8d6fbc1e2c549604c50605da522d59d2385e5f2993 WatchSource:0}: Error finding container 424cd40d6e1937f0ec144b8d6fbc1e2c549604c50605da522d59d2385e5f2993: Status 404 returned error can't find the container with id 424cd40d6e1937f0ec144b8d6fbc1e2c549604c50605da522d59d2385e5f2993
Jul 28 17:24:13.991307 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:13.991226    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2" event={"ID":"90ba8e68-8771-4d00-bba5-96d640379eb7","Type":"ContainerStarted","Data":"424cd40d6e1937f0ec144b8d6fbc1e2c549604c50605da522d59d2385e5f2993"}
Jul 28 17:24:17.005139 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:17.005105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2" event={"ID":"90ba8e68-8771-4d00-bba5-96d640379eb7","Type":"ContainerStarted","Data":"8441dbaa996c79461ac4cd76e644034384313df58d5c54e0ad591e4da56fe23a"}
Jul 28 17:24:17.005570 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:17.005231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:17.023081 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:17.023039    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2" podStartSLOduration=1.5253439420000001 podStartE2EDuration="4.023022521s" totalImagesPullingTime="2.497678579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:24:17.022622083 +0000 UTC m=+457.209215388" watchObservedRunningTime="2026-07-28 17:24:17.023022521 +0000 UTC m=+457.209615825"
Jul 28 17:24:28.011448 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:28.011417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-7k9s2"
Jul 28 17:24:35.190069 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.190022    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6b994f76fc-bhznc" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerName="console" containerID="cri-o://40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8" gracePeriod=15
Jul 28 17:24:35.202079 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.202042    2576 patch_prober.go:28] interesting pod/console-6b994f76fc-bhznc container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused" start-of-body=
Jul 28 17:24:35.202184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.202092    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-6b994f76fc-bhznc" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerName="console" probeResult="failure" output="Get \"https://10.132.0.16:8443/health\": dial tcp 10.132.0.16:8443: connect: connection refused"
Jul 28 17:24:35.430711 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.430684    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:24:35.490124 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490046    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490124 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490114    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h5wzl\" (UniqueName: \"kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490324 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490155    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490324 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490324 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490229    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490324 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490251    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490324 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle\") pod \"49653f39-6de3-4f3f-a5fe-14c43017482d\" (UID: \"49653f39-6de3-4f3f-a5fe-14c43017482d\") "
Jul 28 17:24:35.490582 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490525    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config" (OuterVolumeSpecName: "console-config") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:35.490699 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490668    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:35.490764 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490671    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca" (OuterVolumeSpecName: "service-ca") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:35.490828 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.490799    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:24:35.492367 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.492343    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl" (OuterVolumeSpecName: "kube-api-access-h5wzl") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "kube-api-access-h5wzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:24:35.492481 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.492373    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:35.492481 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.492384    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "49653f39-6de3-4f3f-a5fe-14c43017482d" (UID: "49653f39-6de3-4f3f-a5fe-14c43017482d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:24:35.590889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590851    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-trusted-ca-bundle\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.590889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590886    2576 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-console-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.590889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590896    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5wzl\" (UniqueName: \"kubernetes.io/projected/49653f39-6de3-4f3f-a5fe-14c43017482d-kube-api-access-h5wzl\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.591120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590904    2576 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-service-ca\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.591120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590914    2576 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-serving-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.591120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590923    2576 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/49653f39-6de3-4f3f-a5fe-14c43017482d-oauth-serving-cert\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:35.591120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:35.590931    2576 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/49653f39-6de3-4f3f-a5fe-14c43017482d-console-oauth-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:24:36.073069 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.073037    2576 generic.go:356] "Generic (PLEG): container finished" podID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerID="40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8" exitCode=0
Jul 28 17:24:36.073247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.073084    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6b994f76fc-bhznc"
Jul 28 17:24:36.073247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.073118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b994f76fc-bhznc" event={"ID":"49653f39-6de3-4f3f-a5fe-14c43017482d","Type":"ContainerDied","Data":"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"}
Jul 28 17:24:36.073247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.073159    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6b994f76fc-bhznc" event={"ID":"49653f39-6de3-4f3f-a5fe-14c43017482d","Type":"ContainerDied","Data":"a1ef9162fc251259a78ec861ab26ae2d5c430b2900e5ec741652975a220d3741"}
Jul 28 17:24:36.073247 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.073178    2576 scope.go:117] "RemoveContainer" containerID="40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"
Jul 28 17:24:36.081807 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.081782    2576 scope.go:117] "RemoveContainer" containerID="40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"
Jul 28 17:24:36.082061 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:24:36.082041    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8\": container with ID starting with 40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8 not found: ID does not exist" containerID="40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"
Jul 28 17:24:36.082102 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.082070    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8"} err="failed to get container status \"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8\": rpc error: code = NotFound desc = could not find container \"40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8\": container with ID starting with 40e88cb996acc7d346d12d5253f5f4c64270892b0b439698dfdb960d724001a8 not found: ID does not exist"
Jul 28 17:24:36.095215 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.095192    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:24:36.097151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.097129    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6b994f76fc-bhznc"]
Jul 28 17:24:36.360787 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:24:36.359190    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" path="/var/lib/kubelet/pods/49653f39-6de3-4f3f-a5fe-14c43017482d/volumes"
Jul 28 17:25:40.256538 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:25:40.256469    2576 scope.go:117] "RemoveContainer" containerID="f5c8783b3185f8e7db0879d09ee48cd695ceb3f11106aa2a291c007555d30e08"
Jul 28 17:26:11.769301 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.769263    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:26:11.769794 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.769752    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerName="console"
Jul 28 17:26:11.769794 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.769769    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerName="console"
Jul 28 17:26:11.769867 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.769834    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="49653f39-6de3-4f3f-a5fe-14c43017482d" containerName="console"
Jul 28 17:26:11.772141 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.772124    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.774882 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.774865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cc402-predictor-serving-cert\""
Jul 28 17:26:11.775013 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.774865    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-tdp4m\""
Jul 28 17:26:11.775013 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.774865    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:26:11.776114 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.776098    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-cc402-kube-rbac-proxy-sar-config\""
Jul 28 17:26:11.776192 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.776153    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:26:11.782371 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.782351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:26:11.857354 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.857325    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.857502 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.857392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m8z\" (UniqueName: \"kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.857502 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.857437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.957821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.957792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m8z\" (UniqueName: \"kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.958004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.957840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.958004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.957873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.958676 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.958647    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.960342 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.960319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.964665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.964642    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:26:11.966968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.966948    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:11.968412 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.968389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m8z\" (UniqueName: \"kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z\") pod \"success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") " pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:11.969560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.969527    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cc402-predictor-serving-cert\""
Jul 28 17:26:11.969656 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.969554    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-cc402-kube-rbac-proxy-sar-config\""
Jul 28 17:26:11.976985 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:11.976957    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:26:12.058841 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.058762    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.058841 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.058813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5snnc\" (UniqueName: \"kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.059066 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.058915    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.082731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.082702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:12.160344 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.160313    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.160494 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.160369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5snnc\" (UniqueName: \"kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.160494 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.160427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.160633 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:26:12.160554    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-cc402-predictor-serving-cert: secret "error-404-isvc-cc402-predictor-serving-cert" not found
Jul 28 17:26:12.160633 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:26:12.160625    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls podName:f728a695-7eb1-4d79-94dc-6b7870f877d9 nodeName:}" failed. No retries permitted until 2026-07-28 17:26:12.660599792 +0000 UTC m=+572.847193090 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls") pod "error-404-isvc-cc402-predictor-696c97fc9d-gjdch" (UID: "f728a695-7eb1-4d79-94dc-6b7870f877d9") : secret "error-404-isvc-cc402-predictor-serving-cert" not found
Jul 28 17:26:12.161090 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.161071    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.175109 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.175076    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5snnc\" (UniqueName: \"kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.212743 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.212720    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:26:12.215152 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:26:12.215117    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018dbbed_e2af_491e_8a6c_4b03eccbe7dd.slice/crio-54247c45d5758edc0b6564771a76765e58945a8f6c6ea4e1b8879a09baedc277 WatchSource:0}: Error finding container 54247c45d5758edc0b6564771a76765e58945a8f6c6ea4e1b8879a09baedc277: Status 404 returned error can't find the container with id 54247c45d5758edc0b6564771a76765e58945a8f6c6ea4e1b8879a09baedc277
Jul 28 17:26:12.420744 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.420657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerStarted","Data":"54247c45d5758edc0b6564771a76765e58945a8f6c6ea4e1b8879a09baedc277"}
Jul 28 17:26:12.665196 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.665161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.667734 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.667710    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") pod \"error-404-isvc-cc402-predictor-696c97fc9d-gjdch\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") " pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:12.884043 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:12.884005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:13.083120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:13.083089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:26:13.085836 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:26:13.085313    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf728a695_7eb1_4d79_94dc_6b7870f877d9.slice/crio-13e845ab9297257ca09a28381bba33e821af842c4a407f99f78f34829ea5ce6b WatchSource:0}: Error finding container 13e845ab9297257ca09a28381bba33e821af842c4a407f99f78f34829ea5ce6b: Status 404 returned error can't find the container with id 13e845ab9297257ca09a28381bba33e821af842c4a407f99f78f34829ea5ce6b
Jul 28 17:26:13.429769 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:13.429730    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerStarted","Data":"13e845ab9297257ca09a28381bba33e821af842c4a407f99f78f34829ea5ce6b"}
Jul 28 17:26:28.502999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:28.502960    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerStarted","Data":"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"}
Jul 28 17:26:28.504868 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:28.504837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerStarted","Data":"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"}
Jul 28 17:26:31.521188 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.521150    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerStarted","Data":"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"}
Jul 28 17:26:31.521639 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.521224    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:31.521639 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.521251    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:31.522895 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.522867    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerStarted","Data":"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"}
Jul 28 17:26:31.523064 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.523036    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:26:31.523120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.523069    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:31.523120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.523097    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:31.524150 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.524130    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:26:31.542750 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.542702    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podStartSLOduration=3.041723223 podStartE2EDuration="20.542687461s" totalImagesPullingTime="17.500964238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:31.541551979 +0000 UTC m=+591.728145278" watchObservedRunningTime="2026-07-28 17:26:31.542687461 +0000 UTC m=+591.729280765"
Jul 28 17:26:31.560231 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:31.560182    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podStartSLOduration=2.196715817 podStartE2EDuration="20.560166193s" totalImagesPullingTime="18.363450376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:26:31.559319312 +0000 UTC m=+591.745912627" watchObservedRunningTime="2026-07-28 17:26:31.560166193 +0000 UTC m=+591.746759556"
Jul 28 17:26:32.527188 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:32.527148    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:26:32.527690 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:32.527293    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:26:37.532591 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:37.532559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:26:37.533016 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:37.532991    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:26:37.533086 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:37.533050    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:26:37.533505 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:37.533480    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:26:47.533571 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:47.533514    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:26:47.538175 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:47.533519    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:26:57.533973 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:57.533930    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:26:57.534380 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:26:57.533962    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:27:07.533044 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:07.532963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:27:07.533489 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:07.533458    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:27:17.533713 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:17.533684    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:27:17.534265 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:17.533768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:27:45.850465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.850420    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:27:45.851038 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.850747    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" containerID="cri-o://c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206" gracePeriod=30
Jul 28 17:27:45.851038 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.850842    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kube-rbac-proxy" containerID="cri-o://e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd" gracePeriod=30
Jul 28 17:27:45.901524 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.901487    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:27:45.901760 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.901733    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" containerID="cri-o://fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc" gracePeriod=30
Jul 28 17:27:45.901894 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.901775    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kube-rbac-proxy" containerID="cri-o://67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281" gracePeriod=30
Jul 28 17:27:45.918564 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.918524    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:27:45.921074 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.921054    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:45.924017 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.923897    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-65026-predictor-serving-cert\""
Jul 28 17:27:45.924017 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.923934    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-65026-kube-rbac-proxy-sar-config\""
Jul 28 17:27:45.931601 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:45.931578    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:27:46.005498 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.005461    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:27:46.007760 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.007741    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.010088 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.010059    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-65026-predictor-serving-cert\""
Jul 28 17:27:46.010088 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.010076    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-65026-kube-rbac-proxy-sar-config\""
Jul 28 17:27:46.012411 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.012391    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.012498 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.012431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5ggq\" (UniqueName: \"kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.012591 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.012575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.016749 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.016728    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:27:46.113729 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113646    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.113729 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5ggq\" (UniqueName: \"kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:46.113784    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-65026-predictor-serving-cert: secret "success-200-isvc-65026-predictor-serving-cert" not found
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:46.113863    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls podName:0e987338-9f49-4c49-9b84-cf291a6bd8e4 nodeName:}" failed. No retries permitted until 2026-07-28 17:27:46.613838627 +0000 UTC m=+666.800431921 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls") pod "success-200-isvc-65026-predictor-7ff85d8cb5-7742f" (UID: "0e987338-9f49-4c49-9b84-cf291a6bd8e4") : secret "success-200-isvc-65026-predictor-serving-cert" not found
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113924    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.113957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.113945    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk4fl\" (UniqueName: \"kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.114452 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.114430    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.122121 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.122092    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5ggq\" (UniqueName: \"kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.214637 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.214603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.214807 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.214659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.214807 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.214677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk4fl\" (UniqueName: \"kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.214807 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:46.214776    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-65026-predictor-serving-cert: secret "error-404-isvc-65026-predictor-serving-cert" not found
Jul 28 17:27:46.214950 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:46.214853    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls podName:7317bca5-141f-480b-8e65-1fba4a9b23ab nodeName:}" failed. No retries permitted until 2026-07-28 17:27:46.714833836 +0000 UTC m=+666.901427124 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls") pod "error-404-isvc-65026-predictor-7b8556fc9b-dkthh" (UID: "7317bca5-141f-480b-8e65-1fba4a9b23ab") : secret "error-404-isvc-65026-predictor-serving-cert" not found
Jul 28 17:27:46.215376 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.215354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.223828 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.223801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk4fl\" (UniqueName: \"kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.617850 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.617810    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.620187 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.620167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") pod \"success-200-isvc-65026-predictor-7ff85d8cb5-7742f\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") " pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.718850 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.718815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.721122 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.721095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") pod \"error-404-isvc-65026-predictor-7b8556fc9b-dkthh\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") " pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.802738 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.802707    2576 generic.go:356] "Generic (PLEG): container finished" podID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerID="e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd" exitCode=2
Jul 28 17:27:46.802883 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.802782    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerDied","Data":"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"}
Jul 28 17:27:46.804347 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.804326    2576 generic.go:356] "Generic (PLEG): container finished" podID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerID="67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281" exitCode=2
Jul 28 17:27:46.804453 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.804364    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerDied","Data":"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"}
Jul 28 17:27:46.831752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.831716    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:46.919846 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.919818    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:46.980559 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:46.975939    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:27:47.068606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.068574    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:27:47.072252 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:27:47.072226    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7317bca5_141f_480b_8e65_1fba4a9b23ab.slice/crio-df6783af94a66852aec755cfee99f56321d60b4b790c7a803bba746c78347319 WatchSource:0}: Error finding container df6783af94a66852aec755cfee99f56321d60b4b790c7a803bba746c78347319: Status 404 returned error can't find the container with id df6783af94a66852aec755cfee99f56321d60b4b790c7a803bba746c78347319
Jul 28 17:27:47.527846 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.527810    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.23:8643/healthz\": dial tcp 10.132.0.23:8643: connect: connection refused"
Jul 28 17:27:47.528002 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.527815    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.22:8643/healthz\": dial tcp 10.132.0.22:8643: connect: connection refused"
Jul 28 17:27:47.533373 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.533330    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.22:8080: connect: connection refused"
Jul 28 17:27:47.533538 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.533494    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.23:8080: connect: connection refused"
Jul 28 17:27:47.810267 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.810167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerStarted","Data":"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"}
Jul 28 17:27:47.810267 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.810243    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:47.810504 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.810274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerStarted","Data":"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"}
Jul 28 17:27:47.810504 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.810302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerStarted","Data":"ca381b14fe13ed33e1ebfa1eac045d3c751f7623b4f0d3cc1e86df4131a2db01"}
Jul 28 17:27:47.810504 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.810354    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:47.811747 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.811718    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:27:47.811965 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.811946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerStarted","Data":"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"}
Jul 28 17:27:47.812036 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.811996    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerStarted","Data":"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"}
Jul 28 17:27:47.812036 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.812016    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerStarted","Data":"df6783af94a66852aec755cfee99f56321d60b4b790c7a803bba746c78347319"}
Jul 28 17:27:47.812155 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.812038    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:47.812155 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.812095    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:47.813273 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.813250    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:27:47.830564 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.830496    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podStartSLOduration=2.830479386 podStartE2EDuration="2.830479386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:47.82959405 +0000 UTC m=+668.016187354" watchObservedRunningTime="2026-07-28 17:27:47.830479386 +0000 UTC m=+668.017072691"
Jul 28 17:27:47.848355 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:47.848302    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podStartSLOduration=2.848284418 podStartE2EDuration="2.848284418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:27:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:47.848191036 +0000 UTC m=+668.034784364" watchObservedRunningTime="2026-07-28 17:27:47.848284418 +0000 UTC m=+668.034877725"
Jul 28 17:27:48.815909 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:48.815869    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:27:48.816283 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:48.815969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:27:49.465777 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.465746    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:27:49.512454 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.512432    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:27:49.544536 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544507    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config\") pod \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") "
Jul 28 17:27:49.544536 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544552    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config\") pod \"f728a695-7eb1-4d79-94dc-6b7870f877d9\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") "
Jul 28 17:27:49.544780 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544590    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5snnc\" (UniqueName: \"kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc\") pod \"f728a695-7eb1-4d79-94dc-6b7870f877d9\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") "
Jul 28 17:27:49.544780 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544667    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7m8z\" (UniqueName: \"kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z\") pod \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") "
Jul 28 17:27:49.544780 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544707    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls\") pod \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\" (UID: \"018dbbed-e2af-491e-8a6c-4b03eccbe7dd\") "
Jul 28 17:27:49.544780 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544738    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") pod \"f728a695-7eb1-4d79-94dc-6b7870f877d9\" (UID: \"f728a695-7eb1-4d79-94dc-6b7870f877d9\") "
Jul 28 17:27:49.544983 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544959    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f728a695-7eb1-4d79-94dc-6b7870f877d9" (UID: "f728a695-7eb1-4d79-94dc-6b7870f877d9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:49.545040 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.544962    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "018dbbed-e2af-491e-8a6c-4b03eccbe7dd" (UID: "018dbbed-e2af-491e-8a6c-4b03eccbe7dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:27:49.545094 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.545049    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.545094 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.545069    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.546853 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.546830    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f728a695-7eb1-4d79-94dc-6b7870f877d9" (UID: "f728a695-7eb1-4d79-94dc-6b7870f877d9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:27:49.546952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.546853    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc" (OuterVolumeSpecName: "kube-api-access-5snnc") pod "f728a695-7eb1-4d79-94dc-6b7870f877d9" (UID: "f728a695-7eb1-4d79-94dc-6b7870f877d9"). InnerVolumeSpecName "kube-api-access-5snnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:27:49.546952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.546864    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z" (OuterVolumeSpecName: "kube-api-access-j7m8z") pod "018dbbed-e2af-491e-8a6c-4b03eccbe7dd" (UID: "018dbbed-e2af-491e-8a6c-4b03eccbe7dd"). InnerVolumeSpecName "kube-api-access-j7m8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:27:49.547159 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.547145    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "018dbbed-e2af-491e-8a6c-4b03eccbe7dd" (UID: "018dbbed-e2af-491e-8a6c-4b03eccbe7dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:27:49.646274 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.646242    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5snnc\" (UniqueName: \"kubernetes.io/projected/f728a695-7eb1-4d79-94dc-6b7870f877d9-kube-api-access-5snnc\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.646274 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.646274    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7m8z\" (UniqueName: \"kubernetes.io/projected/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-kube-api-access-j7m8z\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.646460 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.646287    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/018dbbed-e2af-491e-8a6c-4b03eccbe7dd-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.646460 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.646300    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f728a695-7eb1-4d79-94dc-6b7870f877d9-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:27:49.821136 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.821103    2576 generic.go:356] "Generic (PLEG): container finished" podID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerID="fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc" exitCode=0
Jul 28 17:27:49.821597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.821160    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"
Jul 28 17:27:49.821597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.821201    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerDied","Data":"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"}
Jul 28 17:27:49.821597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.821278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch" event={"ID":"f728a695-7eb1-4d79-94dc-6b7870f877d9","Type":"ContainerDied","Data":"13e845ab9297257ca09a28381bba33e821af842c4a407f99f78f34829ea5ce6b"}
Jul 28 17:27:49.821597 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.821308    2576 scope.go:117] "RemoveContainer" containerID="67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"
Jul 28 17:27:49.822984 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.822962    2576 generic.go:356] "Generic (PLEG): container finished" podID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerID="c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206" exitCode=0
Jul 28 17:27:49.823083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.823004    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"
Jul 28 17:27:49.823083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.823046    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerDied","Data":"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"}
Jul 28 17:27:49.823176 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.823092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp" event={"ID":"018dbbed-e2af-491e-8a6c-4b03eccbe7dd","Type":"ContainerDied","Data":"54247c45d5758edc0b6564771a76765e58945a8f6c6ea4e1b8879a09baedc277"}
Jul 28 17:27:49.830101 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.829985    2576 scope.go:117] "RemoveContainer" containerID="fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"
Jul 28 17:27:49.839669 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.839653    2576 scope.go:117] "RemoveContainer" containerID="67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"
Jul 28 17:27:49.839931 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:49.839910    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281\": container with ID starting with 67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281 not found: ID does not exist" containerID="67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"
Jul 28 17:27:49.840007 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.839936    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281"} err="failed to get container status \"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281\": rpc error: code = NotFound desc = could not find container \"67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281\": container with ID starting with 67c3b61514c5ecef34377daf43ad27c9e01174d574ae551d8d5591f2d88b5281 not found: ID does not exist"
Jul 28 17:27:49.840007 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.839953    2576 scope.go:117] "RemoveContainer" containerID="fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"
Jul 28 17:27:49.840197 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:49.840180    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc\": container with ID starting with fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc not found: ID does not exist" containerID="fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"
Jul 28 17:27:49.840233 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.840204    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc"} err="failed to get container status \"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc\": rpc error: code = NotFound desc = could not find container \"fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc\": container with ID starting with fc8a877724e1d67398316181e25832cfd25a6a37c07048955253edb4a6b06dfc not found: ID does not exist"
Jul 28 17:27:49.840233 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.840220    2576 scope.go:117] "RemoveContainer" containerID="e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"
Jul 28 17:27:49.847754 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.847730    2576 scope.go:117] "RemoveContainer" containerID="c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"
Jul 28 17:27:49.849683 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.849662    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:27:49.854298 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.854276    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-cc402-predictor-696c97fc9d-gjdch"]
Jul 28 17:27:49.855116 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.855100    2576 scope.go:117] "RemoveContainer" containerID="e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"
Jul 28 17:27:49.855372 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:49.855353    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd\": container with ID starting with e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd not found: ID does not exist" containerID="e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"
Jul 28 17:27:49.855426 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.855381    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd"} err="failed to get container status \"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd\": rpc error: code = NotFound desc = could not find container \"e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd\": container with ID starting with e3ef40e38d232e8c6d5d4fb3c585b796a133e6c994aac63d0ff483c76ffe28dd not found: ID does not exist"
Jul 28 17:27:49.855426 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.855402    2576 scope.go:117] "RemoveContainer" containerID="c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"
Jul 28 17:27:49.855632 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:27:49.855604    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206\": container with ID starting with c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206 not found: ID does not exist" containerID="c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"
Jul 28 17:27:49.855681 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.855629    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206"} err="failed to get container status \"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206\": rpc error: code = NotFound desc = could not find container \"c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206\": container with ID starting with c16184e401429923960668f90e4412347c361894dd47bce907f9c97c80321206 not found: ID does not exist"
Jul 28 17:27:49.870683 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.870662    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:27:49.877394 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:49.877374    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-cc402-predictor-5ffbc8f9cf-2hxxp"]
Jul 28 17:27:50.357997 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:50.357963    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" path="/var/lib/kubelet/pods/018dbbed-e2af-491e-8a6c-4b03eccbe7dd/volumes"
Jul 28 17:27:50.358378 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:50.358365    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" path="/var/lib/kubelet/pods/f728a695-7eb1-4d79-94dc-6b7870f877d9/volumes"
Jul 28 17:27:53.820230 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:53.820194    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:27:53.820634 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:53.820587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:27:53.820734 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:53.820709    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:27:53.820972 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:27:53.820947    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:28:03.821216 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:03.821173    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:28:03.821705 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:03.821174    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:28:13.821406 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:13.821365    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:28:13.821872 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:13.821373    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:28:23.821321 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:23.821277    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:28:23.821728 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:23.821277    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:28:33.821957 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:33.821875    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:28:33.822370 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:33.822352    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:28:41.863886 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.863850    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864164    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864176    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864188    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864195    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864206    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864213    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864229    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864235    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864282    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864291    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kube-rbac-proxy"
Jul 28 17:28:41.864484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864302    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f728a695-7eb1-4d79-94dc-6b7870f877d9" containerName="kserve-container"
Jul 28 17:28:41.864484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.864309    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="018dbbed-e2af-491e-8a6c-4b03eccbe7dd" containerName="kserve-container"
Jul 28 17:28:41.867200 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.867180    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.867984 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.867962    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.868107 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.867990    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8vd\" (UniqueName: \"kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.868107 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.868037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.869804 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.869773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c20d-predictor-serving-cert\""
Jul 28 17:28:41.869902 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.869774    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8c20d-kube-rbac-proxy-sar-config\""
Jul 28 17:28:41.880630 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.880608    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:28:41.968803 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.968766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.968803 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.968806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8vd\" (UniqueName: \"kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.969074 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.968872    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.969074 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:28:41.968980    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-serving-cert: secret "success-200-isvc-8c20d-predictor-serving-cert" not found
Jul 28 17:28:41.969074 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:28:41.969066    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls podName:12350e64-4ab0-4772-a117-fdccece9f484 nodeName:}" failed. No retries permitted until 2026-07-28 17:28:42.469043268 +0000 UTC m=+722.655636552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls") pod "success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" (UID: "12350e64-4ab0-4772-a117-fdccece9f484") : secret "success-200-isvc-8c20d-predictor-serving-cert" not found
Jul 28 17:28:41.969641 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.969623    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.981236 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.981211    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:28:41.983749 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.983733    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:41.988881 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.988856    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8vd\" (UniqueName: \"kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:41.990116 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.990097    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8c20d-predictor-serving-cert\""
Jul 28 17:28:41.990443 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:41.990426    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8c20d-kube-rbac-proxy-sar-config\""
Jul 28 17:28:42.005231 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.005205    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:28:42.069278 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.069245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.069425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.069285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.069425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.069353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.170484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.170396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.170484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.170430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.170484 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.170460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.170800 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:28:42.170601    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-serving-cert: secret "error-404-isvc-8c20d-predictor-serving-cert" not found
Jul 28 17:28:42.170800 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:28:42.170676    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls podName:a1eb7223-5e8a-4e06-a5d2-36fd81c306d5 nodeName:}" failed. No retries permitted until 2026-07-28 17:28:42.670659185 +0000 UTC m=+722.857252471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls") pod "error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" (UID: "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5") : secret "error-404-isvc-8c20d-predictor-serving-cert" not found
Jul 28 17:28:42.171108 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.171086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.180332 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.180302    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.473639 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.473535    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:42.475931 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.475897    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") pod \"success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") " pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:42.477792 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.477770    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:42.606377 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.606343    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:28:42.609365 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:28:42.609336    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12350e64_4ab0_4772_a117_fdccece9f484.slice/crio-d59ca5a43582b3fcb3327f48c19168980ec94779ea9444eb2a5758fd6537cff9 WatchSource:0}: Error finding container d59ca5a43582b3fcb3327f48c19168980ec94779ea9444eb2a5758fd6537cff9: Status 404 returned error can't find the container with id d59ca5a43582b3fcb3327f48c19168980ec94779ea9444eb2a5758fd6537cff9
Jul 28 17:28:42.675183 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.675157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.678944 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.678913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") pod \"error-404-isvc-8c20d-predictor-5595b88c76-cj8kf\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") " pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:42.900386 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:42.900341    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:43.029120 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.029087    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:28:43.031202 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.031157    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerStarted","Data":"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"}
Jul 28 17:28:43.031330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.031222    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:43.031330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.031253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerStarted","Data":"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"}
Jul 28 17:28:43.031330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.031282    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerStarted","Data":"d59ca5a43582b3fcb3327f48c19168980ec94779ea9444eb2a5758fd6537cff9"}
Jul 28 17:28:43.031330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.031309    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:43.032733 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:28:43.032703    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eb7223_5e8a_4e06_a5d2_36fd81c306d5.slice/crio-1b88b133832e62f7bc5eaeebddc53983e6632de6457fc0a0ba9cb7290a8210ba WatchSource:0}: Error finding container 1b88b133832e62f7bc5eaeebddc53983e6632de6457fc0a0ba9cb7290a8210ba: Status 404 returned error can't find the container with id 1b88b133832e62f7bc5eaeebddc53983e6632de6457fc0a0ba9cb7290a8210ba
Jul 28 17:28:43.032848 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.032825    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:28:43.052019 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:43.051969    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podStartSLOduration=2.05195203 podStartE2EDuration="2.05195203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:43.051149316 +0000 UTC m=+723.237742614" watchObservedRunningTime="2026-07-28 17:28:43.05195203 +0000 UTC m=+723.238545335"
Jul 28 17:28:44.038146 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:44.038106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerStarted","Data":"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"}
Jul 28 17:28:44.038648 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:44.038181    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerStarted","Data":"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"}
Jul 28 17:28:44.038648 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:44.038208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerStarted","Data":"1b88b133832e62f7bc5eaeebddc53983e6632de6457fc0a0ba9cb7290a8210ba"}
Jul 28 17:28:44.038648 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:44.038358    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:28:44.058847 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:44.058800    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podStartSLOduration=3.058786135 podStartE2EDuration="3.058786135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:44.057153503 +0000 UTC m=+724.243746808" watchObservedRunningTime="2026-07-28 17:28:44.058786135 +0000 UTC m=+724.245379439"
Jul 28 17:28:45.043222 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:45.043183    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:45.043595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:45.043255    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:45.044624 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:45.044594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:28:46.046816 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:46.046776    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:28:49.042646 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:49.042617    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:28:49.043073 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:49.043047    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:28:51.051255 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:51.051223    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:28:51.051732 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:51.051704    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:28:59.043558 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:28:59.043501    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:29:01.052187 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:01.052145    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:29:09.043364 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:09.043326    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:29:11.051687 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:11.051648    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:29:19.043700 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:19.043658    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:29:21.051727 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:21.051690    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:8080: connect: connection refused"
Jul 28 17:29:29.044474 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:29.044444    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:29:31.052659 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:29:31.052628    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:37:00.653182 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.653146    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:37:00.653724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.653427    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" containerID="cri-o://cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee" gracePeriod=30
Jul 28 17:37:00.653724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.653474    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kube-rbac-proxy" containerID="cri-o://9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556" gracePeriod=30
Jul 28 17:37:00.742054 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.742022    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:37:00.742277 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.742253    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" containerID="cri-o://76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2" gracePeriod=30
Jul 28 17:37:00.742352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.742274    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kube-rbac-proxy" containerID="cri-o://8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c" gracePeriod=30
Jul 28 17:37:00.778010 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.777975    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:37:00.781158 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.781136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.783584 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.783561    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-59e5e-kube-rbac-proxy-sar-config\""
Jul 28 17:37:00.783808 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.783790    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-59e5e-predictor-serving-cert\""
Jul 28 17:37:00.793417 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.793390    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:37:00.850270 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.850232    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:37:00.853383 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.853364    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:00.856262 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.856235    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-59e5e-kube-rbac-proxy-sar-config\""
Jul 28 17:37:00.856368 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.856315    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-59e5e-predictor-serving-cert\""
Jul 28 17:37:00.864496 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.864471    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:37:00.894534 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.894500    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.894720 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.894575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.894720 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.894637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftk2w\" (UniqueName: \"kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.924719 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.924636    2576 generic.go:356] "Generic (PLEG): container finished" podID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerID="9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556" exitCode=2
Jul 28 17:37:00.924719 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.924694    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerDied","Data":"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"}
Jul 28 17:37:00.926467 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.926436    2576 generic.go:356] "Generic (PLEG): container finished" podID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerID="8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c" exitCode=2
Jul 28 17:37:00.926587 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.926487    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerDied","Data":"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"}
Jul 28 17:37:00.995111 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995065    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jh9bp\" (UniqueName: \"kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:00.995111 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995115    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.995433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:00.995433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995261    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftk2w\" (UniqueName: \"kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.995433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:00.995433 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995368    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.995982 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.995964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:00.997563 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:00.997531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:01.003386 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.003364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftk2w\" (UniqueName: \"kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w\") pod \"success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") " pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:01.091255 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.091215    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:01.096482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.096458    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jh9bp\" (UniqueName: \"kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.096604 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.096502    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.096672 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.096645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.096782 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:01.096765    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-serving-cert: secret "error-404-isvc-59e5e-predictor-serving-cert" not found
Jul 28 17:37:01.096839 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:01.096829    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls podName:88e90ec2-a3f3-45b5-86eb-233b4a64cff3 nodeName:}" failed. No retries permitted until 2026-07-28 17:37:01.596807708 +0000 UTC m=+1221.783401007 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls") pod "error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" (UID: "88e90ec2-a3f3-45b5-86eb-233b4a64cff3") : secret "error-404-isvc-59e5e-predictor-serving-cert" not found
Jul 28 17:37:01.097138 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.097118    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.105908 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.105851    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jh9bp\" (UniqueName: \"kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.224394 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.224238    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:37:01.227052 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:37:01.227027    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad04571_d18c_4f82_a5cf_619bf490310e.slice/crio-582d07ab889aec00206086d1e09c128a495ca0b9ca5265cff3d489c6280f08b1 WatchSource:0}: Error finding container 582d07ab889aec00206086d1e09c128a495ca0b9ca5265cff3d489c6280f08b1: Status 404 returned error can't find the container with id 582d07ab889aec00206086d1e09c128a495ca0b9ca5265cff3d489c6280f08b1
Jul 28 17:37:01.601338 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.601302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.603759 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.603740    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") pod \"error-404-isvc-59e5e-predictor-5468b8979b-jxhfx\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") " pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.764642 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.764602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:01.896056 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.896031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:37:01.898048 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:37:01.898013    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e90ec2_a3f3_45b5_86eb_233b4a64cff3.slice/crio-2427b15383fed4674f587abdf5fb48a7949316ab36390c2b935936acb8368cdc WatchSource:0}: Error finding container 2427b15383fed4674f587abdf5fb48a7949316ab36390c2b935936acb8368cdc: Status 404 returned error can't find the container with id 2427b15383fed4674f587abdf5fb48a7949316ab36390c2b935936acb8368cdc
Jul 28 17:37:01.931561 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.931514    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerStarted","Data":"2427b15383fed4674f587abdf5fb48a7949316ab36390c2b935936acb8368cdc"}
Jul 28 17:37:01.933842 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.933813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerStarted","Data":"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"}
Jul 28 17:37:01.933936 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.933873    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerStarted","Data":"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"}
Jul 28 17:37:01.933936 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.933901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerStarted","Data":"582d07ab889aec00206086d1e09c128a495ca0b9ca5265cff3d489c6280f08b1"}
Jul 28 17:37:01.934641 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.934231    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:01.934641 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.934286    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:01.935809 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.935783    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:01.952091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:01.952043    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podStartSLOduration=1.952026571 podStartE2EDuration="1.952026571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:01.951641041 +0000 UTC m=+1222.138234343" watchObservedRunningTime="2026-07-28 17:37:01.952026571 +0000 UTC m=+1222.138619876"
Jul 28 17:37:02.941606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.941568    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerStarted","Data":"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"}
Jul 28 17:37:02.942046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.941630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerStarted","Data":"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"}
Jul 28 17:37:02.942046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.941817    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:02.942046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.941856    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:02.942046 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.941910    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:02.943142 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.943120    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:02.962112 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:02.962056    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podStartSLOduration=2.962036459 podStartE2EDuration="2.962036459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:02.961520283 +0000 UTC m=+1223.148113600" watchObservedRunningTime="2026-07-28 17:37:02.962036459 +0000 UTC m=+1223.148629767"
Jul 28 17:37:03.816505 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:03.816452    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.25:8643/healthz\": dial tcp 10.132.0.25:8643: connect: connection refused"
Jul 28 17:37:03.816735 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:03.816457    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.24:8643/healthz\": dial tcp 10.132.0.24:8643: connect: connection refused"
Jul 28 17:37:03.820834 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:03.820808    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.25:8080: connect: connection refused"
Jul 28 17:37:03.820984 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:03.820963    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.24:8080: connect: connection refused"
Jul 28 17:37:03.946690 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:03.946657    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:04.105724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.105695    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:37:04.193618 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.193592    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:37:04.223369 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.223337    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config\") pod \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") "
Jul 28 17:37:04.223534 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.223383    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5ggq\" (UniqueName: \"kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq\") pod \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") "
Jul 28 17:37:04.223534 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.223500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") pod \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\" (UID: \"0e987338-9f49-4c49-9b84-cf291a6bd8e4\") "
Jul 28 17:37:04.223747 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.223720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0e987338-9f49-4c49-9b84-cf291a6bd8e4" (UID: "0e987338-9f49-4c49-9b84-cf291a6bd8e4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:04.225489 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.225459    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq" (OuterVolumeSpecName: "kube-api-access-p5ggq") pod "0e987338-9f49-4c49-9b84-cf291a6bd8e4" (UID: "0e987338-9f49-4c49-9b84-cf291a6bd8e4"). InnerVolumeSpecName "kube-api-access-p5ggq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:04.225609 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.225484    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0e987338-9f49-4c49-9b84-cf291a6bd8e4" (UID: "0e987338-9f49-4c49-9b84-cf291a6bd8e4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:04.324990 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.324949    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config\") pod \"7317bca5-141f-480b-8e65-1fba4a9b23ab\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") "
Jul 28 17:37:04.325169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325002    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hk4fl\" (UniqueName: \"kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl\") pod \"7317bca5-141f-480b-8e65-1fba4a9b23ab\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") "
Jul 28 17:37:04.325169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") pod \"7317bca5-141f-480b-8e65-1fba4a9b23ab\" (UID: \"7317bca5-141f-480b-8e65-1fba4a9b23ab\") "
Jul 28 17:37:04.325291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325199    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.325291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325209    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5ggq\" (UniqueName: \"kubernetes.io/projected/0e987338-9f49-4c49-9b84-cf291a6bd8e4-kube-api-access-p5ggq\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.325291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325218    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0e987338-9f49-4c49-9b84-cf291a6bd8e4-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.325432 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.325383    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7317bca5-141f-480b-8e65-1fba4a9b23ab" (UID: "7317bca5-141f-480b-8e65-1fba4a9b23ab"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:04.327147 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.327117    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl" (OuterVolumeSpecName: "kube-api-access-hk4fl") pod "7317bca5-141f-480b-8e65-1fba4a9b23ab" (UID: "7317bca5-141f-480b-8e65-1fba4a9b23ab"). InnerVolumeSpecName "kube-api-access-hk4fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:04.327255 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.327145    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7317bca5-141f-480b-8e65-1fba4a9b23ab" (UID: "7317bca5-141f-480b-8e65-1fba4a9b23ab"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:04.425958 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.425923    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7317bca5-141f-480b-8e65-1fba4a9b23ab-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.425958 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.425957    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.425958 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.425967    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk4fl\" (UniqueName: \"kubernetes.io/projected/7317bca5-141f-480b-8e65-1fba4a9b23ab-kube-api-access-hk4fl\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:37:04.951556 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.951505    2576 generic.go:356] "Generic (PLEG): container finished" podID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerID="cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee" exitCode=0
Jul 28 17:37:04.951969 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.951571    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"
Jul 28 17:37:04.951969 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.951571    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerDied","Data":"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"}
Jul 28 17:37:04.951969 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.951752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f" event={"ID":"0e987338-9f49-4c49-9b84-cf291a6bd8e4","Type":"ContainerDied","Data":"ca381b14fe13ed33e1ebfa1eac045d3c751f7623b4f0d3cc1e86df4131a2db01"}
Jul 28 17:37:04.951969 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.951794    2576 scope.go:117] "RemoveContainer" containerID="9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"
Jul 28 17:37:04.953303 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.953280    2576 generic.go:356] "Generic (PLEG): container finished" podID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerID="76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2" exitCode=0
Jul 28 17:37:04.953401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.953309    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerDied","Data":"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"}
Jul 28 17:37:04.953401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.953331    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"
Jul 28 17:37:04.953401 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.953366    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh" event={"ID":"7317bca5-141f-480b-8e65-1fba4a9b23ab","Type":"ContainerDied","Data":"df6783af94a66852aec755cfee99f56321d60b4b790c7a803bba746c78347319"}
Jul 28 17:37:04.959924 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.959907    2576 scope.go:117] "RemoveContainer" containerID="cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"
Jul 28 17:37:04.967151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.967134    2576 scope.go:117] "RemoveContainer" containerID="9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:04.967802    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556\": container with ID starting with 9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556 not found: ID does not exist" containerID="9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.967838    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556"} err="failed to get container status \"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556\": rpc error: code = NotFound desc = could not find container \"9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556\": container with ID starting with 9276cbb7b6f036153a80b970b40ae93922ca9132eef3767c98d98a767f8aa556 not found: ID does not exist"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.967859    2576 scope.go:117] "RemoveContainer" containerID="cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:04.968114    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee\": container with ID starting with cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee not found: ID does not exist" containerID="cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.968137    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee"} err="failed to get container status \"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee\": rpc error: code = NotFound desc = could not find container \"cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee\": container with ID starting with cfca839d9457c8a59fcda4223e1990293e41f1d8fb1f6215bb5eaf68c570fbee not found: ID does not exist"
Jul 28 17:37:04.969333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.968155    2576 scope.go:117] "RemoveContainer" containerID="8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"
Jul 28 17:37:04.973214 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.973192    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:37:04.976879 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.976860    2576 scope.go:117] "RemoveContainer" containerID="76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"
Jul 28 17:37:04.980348 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.980325    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-65026-predictor-7ff85d8cb5-7742f"]
Jul 28 17:37:04.985167 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.985151    2576 scope.go:117] "RemoveContainer" containerID="8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"
Jul 28 17:37:04.985442 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:04.985423    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c\": container with ID starting with 8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c not found: ID does not exist" containerID="8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"
Jul 28 17:37:04.985488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.985449    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c"} err="failed to get container status \"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c\": rpc error: code = NotFound desc = could not find container \"8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c\": container with ID starting with 8dfa7a00e43121fa5cf0fcabde3ee69313dbbba8eb7df0a20b49f8e09edc7a8c not found: ID does not exist"
Jul 28 17:37:04.985488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.985471    2576 scope.go:117] "RemoveContainer" containerID="76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"
Jul 28 17:37:04.985747 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:04.985727    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2\": container with ID starting with 76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2 not found: ID does not exist" containerID="76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"
Jul 28 17:37:04.985816 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.985756    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2"} err="failed to get container status \"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2\": rpc error: code = NotFound desc = could not find container \"76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2\": container with ID starting with 76f230297d9e4872c50cd70f589cfdfdec47cf5dd02edbb94f97ae6b49b7ace2 not found: ID does not exist"
Jul 28 17:37:04.992521 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.992495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:37:04.996953 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:04.996930    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-65026-predictor-7b8556fc9b-dkthh"]
Jul 28 17:37:06.358091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:06.358061    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" path="/var/lib/kubelet/pods/0e987338-9f49-4c49-9b84-cf291a6bd8e4/volumes"
Jul 28 17:37:06.358537 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:06.358464    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" path="/var/lib/kubelet/pods/7317bca5-141f-480b-8e65-1fba4a9b23ab/volumes"
Jul 28 17:37:07.946925 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:07.946886    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:07.947428 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:07.947405    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:08.950867 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:08.950835    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:08.951333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:08.951307    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:17.948093 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:17.948050    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:18.952033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:18.951995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:27.947633 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:27.947589    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:28.951426 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:28.951379    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:37.948457 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:37.948370    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.28:8080: connect: connection refused"
Jul 28 17:37:38.951762 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:38.951723    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 28 17:37:47.948706 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:47.948669    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:37:48.952703 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:48.952671    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:37:56.659740 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.659705    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:37:56.660202 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.660019    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container" containerID="cri-o://0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2" gracePeriod=30
Jul 28 17:37:56.660202 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.660066    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kube-rbac-proxy" containerID="cri-o://6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d" gracePeriod=30
Jul 28 17:37:56.711045 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.711009    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:37:56.711321 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.711280    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" containerID="cri-o://8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0" gracePeriod=30
Jul 28 17:37:56.711432 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.711359    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kube-rbac-proxy" containerID="cri-o://5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213" gracePeriod=30
Jul 28 17:37:56.759596 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.759538    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:37:56.759949 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.759933    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kube-rbac-proxy"
Jul 28 17:37:56.759992 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.759952    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kube-rbac-proxy"
Jul 28 17:37:56.759992 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.759966    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container"
Jul 28 17:37:56.759992 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.759974    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container"
Jul 28 17:37:56.760092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760002    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kube-rbac-proxy"
Jul 28 17:37:56.760092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760012    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kube-rbac-proxy"
Jul 28 17:37:56.760092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760026    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container"
Jul 28 17:37:56.760092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760035    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container"
Jul 28 17:37:56.760226 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760092    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kube-rbac-proxy"
Jul 28 17:37:56.760226 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760105    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kserve-container"
Jul 28 17:37:56.760226 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760118    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7317bca5-141f-480b-8e65-1fba4a9b23ab" containerName="kserve-container"
Jul 28 17:37:56.760226 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.760126    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e987338-9f49-4c49-9b84-cf291a6bd8e4" containerName="kube-rbac-proxy"
Jul 28 17:37:56.764421 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.764398    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.767049 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.767009    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4108d-predictor-serving-cert\""
Jul 28 17:37:56.767600 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.767074    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4108d-kube-rbac-proxy-sar-config\""
Jul 28 17:37:56.773290 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.773261    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:37:56.866870 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.866835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.867039 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.866986    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5mcn\" (UniqueName: \"kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.867126 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.867103    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.872508 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.872480    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:37:56.875839 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.875820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:56.878184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.878166    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4108d-predictor-serving-cert\""
Jul 28 17:37:56.878280 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.878209    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4108d-kube-rbac-proxy-sar-config\""
Jul 28 17:37:56.884464 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.884444    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:37:56.967696 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967616    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l4z\" (UniqueName: \"kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:56.967696 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967669    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.967874 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:56.967874 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.967874 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5mcn\" (UniqueName: \"kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.967996 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.967895    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:56.968380 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.968354    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.970170 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.970149    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:56.976083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:56.976061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5mcn\" (UniqueName: \"kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn\") pod \"success-200-isvc-4108d-predictor-6fdcc84849-ntwgz\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") " pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:57.068456 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.068419    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l4z\" (UniqueName: \"kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.068685 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.068475    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.068685 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.068604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.068804 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:57.068724    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-4108d-predictor-serving-cert: secret "error-404-isvc-4108d-predictor-serving-cert" not found
Jul 28 17:37:57.068804 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:37:57.068801    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls podName:2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16 nodeName:}" failed. No retries permitted until 2026-07-28 17:37:57.568779456 +0000 UTC m=+1277.755372747 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls") pod "error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" (UID: "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16") : secret "error-404-isvc-4108d-predictor-serving-cert" not found
Jul 28 17:37:57.069152 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.069126    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.075761 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.075740    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:57.077861 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.077841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l4z\" (UniqueName: \"kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.170114 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.170076    2576 generic.go:356] "Generic (PLEG): container finished" podID="12350e64-4ab0-4772-a117-fdccece9f484" containerID="5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213" exitCode=2
Jul 28 17:37:57.170306 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.170143    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerDied","Data":"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"}
Jul 28 17:37:57.172035 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.172012    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerID="6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d" exitCode=2
Jul 28 17:37:57.172152 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.172036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerDied","Data":"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"}
Jul 28 17:37:57.226586 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.226559    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:37:57.228886 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:37:57.228861    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0c82467_5c68_4d9e_bf2f_599d5c8e37d3.slice/crio-d937566c0d737d18c3acdc3c68d19286d85a458ea2cffcb4a7055ca70bcb3bba WatchSource:0}: Error finding container d937566c0d737d18c3acdc3c68d19286d85a458ea2cffcb4a7055ca70bcb3bba: Status 404 returned error can't find the container with id d937566c0d737d18c3acdc3c68d19286d85a458ea2cffcb4a7055ca70bcb3bba
Jul 28 17:37:57.571999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.571964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.574335 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.574310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") pod \"error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") " pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.787017 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.786965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:57.917907 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:57.917883    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:37:57.920335 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:37:57.920306    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7ef913_e7c5_4a2c_9b8a_1b2ec72cfe16.slice/crio-f7199a05ea0da2d49675296b4b2ca66ba165d4834242e978f74fb8ce57b15a1e WatchSource:0}: Error finding container f7199a05ea0da2d49675296b4b2ca66ba165d4834242e978f74fb8ce57b15a1e: Status 404 returned error can't find the container with id f7199a05ea0da2d49675296b4b2ca66ba165d4834242e978f74fb8ce57b15a1e
Jul 28 17:37:58.178106 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.178006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerStarted","Data":"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"}
Jul 28 17:37:58.178106 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.178075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerStarted","Data":"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"}
Jul 28 17:37:58.178106 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.178104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerStarted","Data":"d937566c0d737d18c3acdc3c68d19286d85a458ea2cffcb4a7055ca70bcb3bba"}
Jul 28 17:37:58.178367 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.178138    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:58.179745 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.179720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerStarted","Data":"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"}
Jul 28 17:37:58.179894 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.179768    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerStarted","Data":"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"}
Jul 28 17:37:58.179894 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.179787    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerStarted","Data":"f7199a05ea0da2d49675296b4b2ca66ba165d4834242e978f74fb8ce57b15a1e"}
Jul 28 17:37:58.179894 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.179810    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:58.179894 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.179858    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:37:58.181440 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.181413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:37:58.196523 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.196477    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podStartSLOduration=2.196462436 podStartE2EDuration="2.196462436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:58.195018625 +0000 UTC m=+1278.381611930" watchObservedRunningTime="2026-07-28 17:37:58.196462436 +0000 UTC m=+1278.383055740"
Jul 28 17:37:58.212611 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:58.212567    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podStartSLOduration=2.212537326 podStartE2EDuration="2.212537326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:58.211527248 +0000 UTC m=+1278.398120577" watchObservedRunningTime="2026-07-28 17:37:58.212537326 +0000 UTC m=+1278.399130688"
Jul 28 17:37:59.039413 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:59.039364    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 28 17:37:59.043694 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:59.043662    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 28 17:37:59.183850 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:59.183810    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:37:59.183850 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:59.183837    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:37:59.184925 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:37:59.184902    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:00.180612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.180584    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:38:00.183165 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.183143    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:38:00.194645 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.194570    2576 generic.go:356] "Generic (PLEG): container finished" podID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerID="0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2" exitCode=0
Jul 28 17:38:00.194645 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.194630    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"
Jul 28 17:38:00.194825 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.194654    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerDied","Data":"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"}
Jul 28 17:38:00.194825 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.194748    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf" event={"ID":"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5","Type":"ContainerDied","Data":"1b88b133832e62f7bc5eaeebddc53983e6632de6457fc0a0ba9cb7290a8210ba"}
Jul 28 17:38:00.194825 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.194787    2576 scope.go:117] "RemoveContainer" containerID="6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"
Jul 28 17:38:00.196067 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.196050    2576 generic.go:356] "Generic (PLEG): container finished" podID="12350e64-4ab0-4772-a117-fdccece9f484" containerID="8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0" exitCode=0
Jul 28 17:38:00.196171 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.196114    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"
Jul 28 17:38:00.196171 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.196121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerDied","Data":"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"}
Jul 28 17:38:00.196278 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.196184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq" event={"ID":"12350e64-4ab0-4772-a117-fdccece9f484","Type":"ContainerDied","Data":"d59ca5a43582b3fcb3327f48c19168980ec94779ea9444eb2a5758fd6537cff9"}
Jul 28 17:38:00.196599 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.196572    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:00.204276 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.204258    2576 scope.go:117] "RemoveContainer" containerID="0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"
Jul 28 17:38:00.211993 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.211973    2576 scope.go:117] "RemoveContainer" containerID="6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"
Jul 28 17:38:00.212247 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:00.212227    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d\": container with ID starting with 6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d not found: ID does not exist" containerID="6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"
Jul 28 17:38:00.212305 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.212256    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d"} err="failed to get container status \"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d\": rpc error: code = NotFound desc = could not find container \"6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d\": container with ID starting with 6417bc483ae205e028130e1d65b56ef8949a5a77d72ee7bef1d2975c20d3802d not found: ID does not exist"
Jul 28 17:38:00.212305 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.212273    2576 scope.go:117] "RemoveContainer" containerID="0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"
Jul 28 17:38:00.212535 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:00.212521    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2\": container with ID starting with 0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2 not found: ID does not exist" containerID="0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"
Jul 28 17:38:00.212622 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.212563    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2"} err="failed to get container status \"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2\": rpc error: code = NotFound desc = could not find container \"0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2\": container with ID starting with 0cf42c635bc5c0c2dbcf8429cdb154bd45abb2e3a490e448d296d66cb54859c2 not found: ID does not exist"
Jul 28 17:38:00.212622 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.212584    2576 scope.go:117] "RemoveContainer" containerID="5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"
Jul 28 17:38:00.221629 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.221606    2576 scope.go:117] "RemoveContainer" containerID="8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"
Jul 28 17:38:00.229365 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.229337    2576 scope.go:117] "RemoveContainer" containerID="5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"
Jul 28 17:38:00.229652 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:00.229627    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213\": container with ID starting with 5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213 not found: ID does not exist" containerID="5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"
Jul 28 17:38:00.229731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.229659    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213"} err="failed to get container status \"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213\": rpc error: code = NotFound desc = could not find container \"5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213\": container with ID starting with 5cd3402bb9802e8f49ac15e028f7ed538aee4dd5f3c30b0b02135b525ed6d213 not found: ID does not exist"
Jul 28 17:38:00.229731 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.229684    2576 scope.go:117] "RemoveContainer" containerID="8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"
Jul 28 17:38:00.229937 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:00.229920    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0\": container with ID starting with 8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0 not found: ID does not exist" containerID="8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"
Jul 28 17:38:00.230001 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.229948    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0"} err="failed to get container status \"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0\": rpc error: code = NotFound desc = could not find container \"8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0\": container with ID starting with 8d45233a9d4b217f48191c3ff205059d52da1ea16ba5aaad95dc27fca28f96c0 not found: ID does not exist"
Jul 28 17:38:00.294005 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.293966    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg8vd\" (UniqueName: \"kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd\") pod \"12350e64-4ab0-4772-a117-fdccece9f484\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") "
Jul 28 17:38:00.294186 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config\") pod \"12350e64-4ab0-4772-a117-fdccece9f484\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") "
Jul 28 17:38:00.294186 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294051    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") pod \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") "
Jul 28 17:38:00.294186 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294105    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6\") pod \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") "
Jul 28 17:38:00.294186 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294128    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") pod \"12350e64-4ab0-4772-a117-fdccece9f484\" (UID: \"12350e64-4ab0-4772-a117-fdccece9f484\") "
Jul 28 17:38:00.294186 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294161    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config\") pod \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\" (UID: \"a1eb7223-5e8a-4e06-a5d2-36fd81c306d5\") "
Jul 28 17:38:00.294451 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294423    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "12350e64-4ab0-4772-a117-fdccece9f484" (UID: "12350e64-4ab0-4772-a117-fdccece9f484"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:00.294512 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294495    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/12350e64-4ab0-4772-a117-fdccece9f484-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.294572 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.294512    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" (UID: "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:00.296617 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.296589    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "12350e64-4ab0-4772-a117-fdccece9f484" (UID: "12350e64-4ab0-4772-a117-fdccece9f484"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:00.296710 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.296619    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" (UID: "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:00.296710 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.296673    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6" (OuterVolumeSpecName: "kube-api-access-ks6h6") pod "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" (UID: "a1eb7223-5e8a-4e06-a5d2-36fd81c306d5"). InnerVolumeSpecName "kube-api-access-ks6h6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:00.296710 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.296696    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd" (OuterVolumeSpecName: "kube-api-access-tg8vd") pod "12350e64-4ab0-4772-a117-fdccece9f484" (UID: "12350e64-4ab0-4772-a117-fdccece9f484"). InnerVolumeSpecName "kube-api-access-tg8vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:00.394929 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.394893    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8vd\" (UniqueName: \"kubernetes.io/projected/12350e64-4ab0-4772-a117-fdccece9f484-kube-api-access-tg8vd\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.394929 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.394927    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.395091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.394939    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ks6h6\" (UniqueName: \"kubernetes.io/projected/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-api-access-ks6h6\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.395091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.394947    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/12350e64-4ab0-4772-a117-fdccece9f484-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.395091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.394956    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:00.513365 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.513328    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:38:00.519651 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.519627    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8c20d-predictor-5595b88c76-cj8kf"]
Jul 28 17:38:00.532913 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.532870    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:38:00.536867 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:00.536838    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8c20d-predictor-f89cb6ff7-8cwdq"]
Jul 28 17:38:02.357698 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:02.357666    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12350e64-4ab0-4772-a117-fdccece9f484" path="/var/lib/kubelet/pods/12350e64-4ab0-4772-a117-fdccece9f484/volumes"
Jul 28 17:38:02.358099 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:02.358085    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" path="/var/lib/kubelet/pods/a1eb7223-5e8a-4e06-a5d2-36fd81c306d5/volumes"
Jul 28 17:38:04.188041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:04.188011    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:38:04.188507 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:04.188480    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:38:05.201747 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:05.201717    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:38:05.202207 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:05.202183    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:10.972886 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:10.972795    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:38:10.973796 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:10.973738    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container" containerID="cri-o://c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a" gracePeriod=30
Jul 28 17:38:10.974211 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:10.974162    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kube-rbac-proxy" containerID="cri-o://da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397" gracePeriod=30
Jul 28 17:38:11.029853 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.029817    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:38:11.030129 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.030072    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container" containerID="cri-o://342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a" gracePeriod=30
Jul 28 17:38:11.030129 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.030103    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kube-rbac-proxy" containerID="cri-o://4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20" gracePeriod=30
Jul 28 17:38:11.065151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065116    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:38:11.065441 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065429    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kube-rbac-proxy"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065442    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kube-rbac-proxy"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065459    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065464    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065473    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kube-rbac-proxy"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065479    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kube-rbac-proxy"
Jul 28 17:38:11.065490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065488    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container"
Jul 28 17:38:11.065695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065493    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container"
Jul 28 17:38:11.065695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065564    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kube-rbac-proxy"
Jul 28 17:38:11.065695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065574    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="12350e64-4ab0-4772-a117-fdccece9f484" containerName="kserve-container"
Jul 28 17:38:11.065695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065583    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kserve-container"
Jul 28 17:38:11.065695 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.065589    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1eb7223-5e8a-4e06-a5d2-36fd81c306d5" containerName="kube-rbac-proxy"
Jul 28 17:38:11.068633 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.068614    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.071069 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.071047    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c57a1-kube-rbac-proxy-sar-config\""
Jul 28 17:38:11.071168 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.071149    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c57a1-predictor-serving-cert\""
Jul 28 17:38:11.080095 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.080070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:38:11.165752 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.165723    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:38:11.169042 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.169021    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.171335 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.171310    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c57a1-kube-rbac-proxy-sar-config\""
Jul 28 17:38:11.171335 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.171340    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c57a1-predictor-serving-cert\""
Jul 28 17:38:11.182175 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.182149    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:38:11.192515 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.192485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfbsn\" (UniqueName: \"kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.192665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.192558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.192665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.192648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.245510 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.245415    2576 generic.go:356] "Generic (PLEG): container finished" podID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerID="da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397" exitCode=2
Jul 28 17:38:11.245510 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.245491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerDied","Data":"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"}
Jul 28 17:38:11.247097 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.247075    2576 generic.go:356] "Generic (PLEG): container finished" podID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerID="4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20" exitCode=2
Jul 28 17:38:11.247210 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.247106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerDied","Data":"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"}
Jul 28 17:38:11.293508 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293473    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hnqx\" (UniqueName: \"kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.293508 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.293756 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfbsn\" (UniqueName: \"kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.293756 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293597    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.293756 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.293756 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.293750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.294167 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.294141    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.296166 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.296132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.301616 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.301590    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfbsn\" (UniqueName: \"kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn\") pod \"success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") " pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.378893 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.378859    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:11.395333 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.395297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.395488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.395350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.395488 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.395420    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hnqx\" (UniqueName: \"kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.395990 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.395967    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.397668 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.397651    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.404196 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.404172    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hnqx\" (UniqueName: \"kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx\") pod \"error-404-isvc-c57a1-predictor-9bd97b66d-twn7s\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") " pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.478952 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.478906    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:11.517368 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.517341    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:38:11.518826 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:38:11.518799    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda377c3f4_672e_4217_9efd_cbd90e225034.slice/crio-86b3efe50fe3cc8b0356b427cd0350dc7f8d2fac9969992f9ba49150f8ed31d4 WatchSource:0}: Error finding container 86b3efe50fe3cc8b0356b427cd0350dc7f8d2fac9969992f9ba49150f8ed31d4: Status 404 returned error can't find the container with id 86b3efe50fe3cc8b0356b427cd0350dc7f8d2fac9969992f9ba49150f8ed31d4
Jul 28 17:38:11.620881 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:11.620786    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:38:11.629648 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:38:11.629620    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45c665a0_4abc_4471_905e_b157af3578c5.slice/crio-c1ffb9f1eda23a30a42fe3ffb37e142a69f76f3f0382b26feb19615bcbb87079 WatchSource:0}: Error finding container c1ffb9f1eda23a30a42fe3ffb37e142a69f76f3f0382b26feb19615bcbb87079: Status 404 returned error can't find the container with id c1ffb9f1eda23a30a42fe3ffb37e142a69f76f3f0382b26feb19615bcbb87079
Jul 28 17:38:12.254485 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.254447    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerStarted","Data":"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"}
Jul 28 17:38:12.254968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.254506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerStarted","Data":"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"}
Jul 28 17:38:12.254968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.254531    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerStarted","Data":"86b3efe50fe3cc8b0356b427cd0350dc7f8d2fac9969992f9ba49150f8ed31d4"}
Jul 28 17:38:12.254968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.254589    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:12.254968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.254623    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:12.256329 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256303    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerStarted","Data":"99840d13833df9aafc954eba17468cd984e2af282c3c5eabb108b5785468c46a"}
Jul 28 17:38:12.256423 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256364    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:12.256423 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256374    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:12.256423 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256397    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:12.256533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256422    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerStarted","Data":"8751c1d14792063f56c6cec82aedaaceeb999840d93c4e914f3853eef4d6273c"}
Jul 28 17:38:12.256533 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.256449    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerStarted","Data":"c1ffb9f1eda23a30a42fe3ffb37e142a69f76f3f0382b26feb19615bcbb87079"}
Jul 28 17:38:12.257513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.257487    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:12.273694 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.273651    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podStartSLOduration=1.273636607 podStartE2EDuration="1.273636607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:12.271958633 +0000 UTC m=+1292.458551949" watchObservedRunningTime="2026-07-28 17:38:12.273636607 +0000 UTC m=+1292.460229911"
Jul 28 17:38:12.292297 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.292249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podStartSLOduration=1.292232271 podStartE2EDuration="1.292232271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:12.29137732 +0000 UTC m=+1292.477970638" watchObservedRunningTime="2026-07-28 17:38:12.292232271 +0000 UTC m=+1292.478825575"
Jul 28 17:38:12.942581 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:12.942516    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.28:8643/healthz\": dial tcp 10.132.0.28:8643: connect: connection refused"
Jul 28 17:38:13.260912 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:13.260867    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:13.261292 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:13.260974    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:13.952172 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:13.952104    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 28 17:38:14.189284 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.189244    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:38:14.584338 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.584313    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:38:14.623182 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.623096    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config\") pod \"4ad04571-d18c-4f82-a5cf-619bf490310e\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") "
Jul 28 17:38:14.623182 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.623136    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftk2w\" (UniqueName: \"kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w\") pod \"4ad04571-d18c-4f82-a5cf-619bf490310e\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") "
Jul 28 17:38:14.623182 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.623165    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls\") pod \"4ad04571-d18c-4f82-a5cf-619bf490310e\" (UID: \"4ad04571-d18c-4f82-a5cf-619bf490310e\") "
Jul 28 17:38:14.623538 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.623508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4ad04571-d18c-4f82-a5cf-619bf490310e" (UID: "4ad04571-d18c-4f82-a5cf-619bf490310e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:14.625264 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.625238    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4ad04571-d18c-4f82-a5cf-619bf490310e" (UID: "4ad04571-d18c-4f82-a5cf-619bf490310e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:14.625367 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.625298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w" (OuterVolumeSpecName: "kube-api-access-ftk2w") pod "4ad04571-d18c-4f82-a5cf-619bf490310e" (UID: "4ad04571-d18c-4f82-a5cf-619bf490310e"). InnerVolumeSpecName "kube-api-access-ftk2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:14.724241 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.724202    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:14.724241 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.724237    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftk2w\" (UniqueName: \"kubernetes.io/projected/4ad04571-d18c-4f82-a5cf-619bf490310e-kube-api-access-ftk2w\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:14.724241 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:14.724247    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4ad04571-d18c-4f82-a5cf-619bf490310e-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:15.030403 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.030378    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:38:15.127465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.127426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jh9bp\" (UniqueName: \"kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp\") pod \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") "
Jul 28 17:38:15.127665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.127498    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") pod \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") "
Jul 28 17:38:15.127665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.127623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config\") pod \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\" (UID: \"88e90ec2-a3f3-45b5-86eb-233b4a64cff3\") "
Jul 28 17:38:15.127974 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.127954    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88e90ec2-a3f3-45b5-86eb-233b4a64cff3" (UID: "88e90ec2-a3f3-45b5-86eb-233b4a64cff3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:15.129521 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.129493    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88e90ec2-a3f3-45b5-86eb-233b4a64cff3" (UID: "88e90ec2-a3f3-45b5-86eb-233b4a64cff3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:15.129666 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.129527    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp" (OuterVolumeSpecName: "kube-api-access-jh9bp") pod "88e90ec2-a3f3-45b5-86eb-233b4a64cff3" (UID: "88e90ec2-a3f3-45b5-86eb-233b4a64cff3"). InnerVolumeSpecName "kube-api-access-jh9bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:15.202985 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.202895    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:15.228683 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.228656    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:15.228683 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.228682    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jh9bp\" (UniqueName: \"kubernetes.io/projected/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-kube-api-access-jh9bp\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:15.228796 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.228693    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88e90ec2-a3f3-45b5-86eb-233b4a64cff3-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:38:15.270697 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.270658    2576 generic.go:356] "Generic (PLEG): container finished" podID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerID="c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a" exitCode=0
Jul 28 17:38:15.270851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.270716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"
Jul 28 17:38:15.270851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.270741    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerDied","Data":"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"}
Jul 28 17:38:15.270851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.270796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx" event={"ID":"88e90ec2-a3f3-45b5-86eb-233b4a64cff3","Type":"ContainerDied","Data":"2427b15383fed4674f587abdf5fb48a7949316ab36390c2b935936acb8368cdc"}
Jul 28 17:38:15.270851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.270825    2576 scope.go:117] "RemoveContainer" containerID="da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"
Jul 28 17:38:15.272435 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.272406    2576 generic.go:356] "Generic (PLEG): container finished" podID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerID="342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a" exitCode=0
Jul 28 17:38:15.272600 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.272456    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"
Jul 28 17:38:15.272600 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.272461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerDied","Data":"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"}
Jul 28 17:38:15.272600 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.272517    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb" event={"ID":"4ad04571-d18c-4f82-a5cf-619bf490310e","Type":"ContainerDied","Data":"582d07ab889aec00206086d1e09c128a495ca0b9ca5265cff3d489c6280f08b1"}
Jul 28 17:38:15.280425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.280397    2576 scope.go:117] "RemoveContainer" containerID="c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"
Jul 28 17:38:15.288410 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.288392    2576 scope.go:117] "RemoveContainer" containerID="da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"
Jul 28 17:38:15.288700 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:15.288675    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397\": container with ID starting with da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397 not found: ID does not exist" containerID="da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"
Jul 28 17:38:15.288751 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.288713    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397"} err="failed to get container status \"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397\": rpc error: code = NotFound desc = could not find container \"da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397\": container with ID starting with da0fdb7a42198a55a2b95aee5a8f315fab4583381e61020da5e023cc670d7397 not found: ID does not exist"
Jul 28 17:38:15.288751 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.288734    2576 scope.go:117] "RemoveContainer" containerID="c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"
Jul 28 17:38:15.289002 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:15.288984    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a\": container with ID starting with c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a not found: ID does not exist" containerID="c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"
Jul 28 17:38:15.289079 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.289013    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a"} err="failed to get container status \"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a\": rpc error: code = NotFound desc = could not find container \"c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a\": container with ID starting with c969329211bc8c7a0c7b16e0d27e1ee8213471763d767a84ade6ac53de41f05a not found: ID does not exist"
Jul 28 17:38:15.289079 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.289036    2576 scope.go:117] "RemoveContainer" containerID="4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"
Jul 28 17:38:15.296800 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.296763    2576 scope.go:117] "RemoveContainer" containerID="342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"
Jul 28 17:38:15.298259 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.298239    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:38:15.302233 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.302209    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-59e5e-predictor-5984bd5fb5-xqkfb"]
Jul 28 17:38:15.304516 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.304499    2576 scope.go:117] "RemoveContainer" containerID="4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"
Jul 28 17:38:15.304788 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:15.304767    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20\": container with ID starting with 4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20 not found: ID does not exist" containerID="4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"
Jul 28 17:38:15.304863 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.304797    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20"} err="failed to get container status \"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20\": rpc error: code = NotFound desc = could not find container \"4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20\": container with ID starting with 4b1d7e67e793b7727b78ee2c08f431c64c0a2770e7c1ca346e4d2a3a13bffb20 not found: ID does not exist"
Jul 28 17:38:15.304863 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.304820    2576 scope.go:117] "RemoveContainer" containerID="342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"
Jul 28 17:38:15.305079 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:38:15.305064    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a\": container with ID starting with 342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a not found: ID does not exist" containerID="342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"
Jul 28 17:38:15.305119 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.305086    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a"} err="failed to get container status \"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a\": rpc error: code = NotFound desc = could not find container \"342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a\": container with ID starting with 342b9808a952da3fd6b7e4dabd14bd09aa7a457fbdb67858c5432685b5ff7b6a not found: ID does not exist"
Jul 28 17:38:15.313289 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.313266    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:38:15.317409 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:15.317389    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-59e5e-predictor-5468b8979b-jxhfx"]
Jul 28 17:38:16.357628 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:16.357600    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" path="/var/lib/kubelet/pods/4ad04571-d18c-4f82-a5cf-619bf490310e/volumes"
Jul 28 17:38:16.358041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:16.358027    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" path="/var/lib/kubelet/pods/88e90ec2-a3f3-45b5-86eb-233b4a64cff3/volumes"
Jul 28 17:38:18.265426 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:18.265396    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:38:18.265888 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:18.265561    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:38:18.265888 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:18.265806    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:18.266267 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:18.266243    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:24.188560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:24.188495    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:38:25.202968 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:25.202919    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:28.266452 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:28.266413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:28.266853 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:28.266413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:34.189357 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:34.189317    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:38:35.202637 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:35.202588    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 28 17:38:38.266205 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:38.266160    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:38.266630 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:38.266326    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:44.188999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:44.188958    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.31:8080: connect: connection refused"
Jul 28 17:38:45.203748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:45.203716    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:38:48.266411 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:48.266369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:38:48.266787 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:48.266369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:54.189447 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:54.189415    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:38:58.266617 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:58.266567    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:38:58.267423 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:38:58.267405    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:39:07.001490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.001377    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:39:07.001994 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.001684    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container" containerID="cri-o://641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb" gracePeriod=30
Jul 28 17:39:07.001994 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.001730    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kube-rbac-proxy" containerID="cri-o://f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e" gracePeriod=30
Jul 28 17:39:07.051612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.051568    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:39:07.051893 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.051827    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container" containerID="cri-o://5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512" gracePeriod=30
Jul 28 17:39:07.052019 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.051910    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kube-rbac-proxy" containerID="cri-o://1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5" gracePeriod=30
Jul 28 17:39:07.071398 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071371    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:39:07.071738 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071724    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071740    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071751    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kube-rbac-proxy"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071756    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kube-rbac-proxy"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071766    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071771    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071782    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kube-rbac-proxy"
Jul 28 17:39:07.071799 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071787    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kube-rbac-proxy"
Jul 28 17:39:07.072004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071833    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kserve-container"
Jul 28 17:39:07.072004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071843    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kube-rbac-proxy"
Jul 28 17:39:07.072004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071849    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ad04571-d18c-4f82-a5cf-619bf490310e" containerName="kserve-container"
Jul 28 17:39:07.072004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.071856    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e90ec2-a3f3-45b5-86eb-233b4a64cff3" containerName="kube-rbac-proxy"
Jul 28 17:39:07.076357 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.076337    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.078623 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.078602    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7add-predictor-serving-cert\""
Jul 28 17:39:07.078767 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.078749    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c7add-kube-rbac-proxy-sar-config\""
Jul 28 17:39:07.090180 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.090147    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:39:07.158909 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.158880    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:39:07.162209 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.162189    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.164632 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.164605    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7add-predictor-serving-cert\""
Jul 28 17:39:07.164728 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.164661    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c7add-kube-rbac-proxy-sar-config\""
Jul 28 17:39:07.172045 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.172022    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:39:07.252995 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.252966    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qql7\" (UniqueName: \"kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.253159 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.253006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.253159 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.253027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrk7\" (UniqueName: \"kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.253159 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.253142    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.253320 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.253206    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.253320 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.253228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.354386 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.354386 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354385    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.354661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qql7\" (UniqueName: \"kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.354661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354533    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.354661 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrk7\" (UniqueName: \"kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.354814 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.354691    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.354875 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:07.354824    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c7add-predictor-serving-cert: secret "error-404-isvc-c7add-predictor-serving-cert" not found
Jul 28 17:39:07.354937 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:07.354894    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls podName:d8a795a5-5625-4829-8306-0fc96a2377dd nodeName:}" failed. No retries permitted until 2026-07-28 17:39:07.854874397 +0000 UTC m=+1348.041467691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls") pod "error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" (UID: "d8a795a5-5625-4829-8306-0fc96a2377dd") : secret "error-404-isvc-c7add-predictor-serving-cert" not found
Jul 28 17:39:07.355238 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.355220    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.355288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.355268    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.357085 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.357062    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.366229 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.366201    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qql7\" (UniqueName: \"kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.366328 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.366263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrk7\" (UniqueName: \"kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7\") pod \"success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") " pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.393260 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.393222    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:07.495145 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.495114    2576 generic.go:356] "Generic (PLEG): container finished" podID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerID="f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e" exitCode=2
Jul 28 17:39:07.495303 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.495197    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerDied","Data":"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"}
Jul 28 17:39:07.496632 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.496602    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerID="1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5" exitCode=2
Jul 28 17:39:07.496738 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.496677    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerDied","Data":"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"}
Jul 28 17:39:07.550307 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.550226    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:39:07.553833 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:39:07.553808    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3668a6c_7b0b_4814_9ced_5f29d4f88ad1.slice/crio-b568d977196833ac35db9fbff4644584001f45b7f5fba80c0af0d639368125df WatchSource:0}: Error finding container b568d977196833ac35db9fbff4644584001f45b7f5fba80c0af0d639368125df: Status 404 returned error can't find the container with id b568d977196833ac35db9fbff4644584001f45b7f5fba80c0af0d639368125df
Jul 28 17:39:07.858480 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.858393    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:07.860777 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:07.860755    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") pod \"error-404-isvc-c7add-predictor-6d685ddd8-cd8kt\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") " pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:08.073514 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.073472    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:08.200775 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.200616    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:39:08.203408 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:39:08.203382    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8a795a5_5625_4829_8306_0fc96a2377dd.slice/crio-9c1df0d3d471dbe8a5202ea60b4074b87b290074356a08a42c8e40470a652d36 WatchSource:0}: Error finding container 9c1df0d3d471dbe8a5202ea60b4074b87b290074356a08a42c8e40470a652d36: Status 404 returned error can't find the container with id 9c1df0d3d471dbe8a5202ea60b4074b87b290074356a08a42c8e40470a652d36
Jul 28 17:39:08.267456 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.267434    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:39:08.502984 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.502952    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerStarted","Data":"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"}
Jul 28 17:39:08.503173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.503020    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:08.503173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.503054    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerStarted","Data":"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"}
Jul 28 17:39:08.503173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.503085    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerStarted","Data":"b568d977196833ac35db9fbff4644584001f45b7f5fba80c0af0d639368125df"}
Jul 28 17:39:08.504822 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.504797    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerStarted","Data":"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"}
Jul 28 17:39:08.504933 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.504857    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerStarted","Data":"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"}
Jul 28 17:39:08.504933 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.504890    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerStarted","Data":"9c1df0d3d471dbe8a5202ea60b4074b87b290074356a08a42c8e40470a652d36"}
Jul 28 17:39:08.505094 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.505078    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:08.505148 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.505118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:08.506815 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.506785    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:08.521446 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.521401    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podStartSLOduration=1.521386103 podStartE2EDuration="1.521386103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:08.519931849 +0000 UTC m=+1348.706525192" watchObservedRunningTime="2026-07-28 17:39:08.521386103 +0000 UTC m=+1348.707979407"
Jul 28 17:39:08.537012 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:08.536946    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podStartSLOduration=1.536924413 podStartE2EDuration="1.536924413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:39:08.535814475 +0000 UTC m=+1348.722407886" watchObservedRunningTime="2026-07-28 17:39:08.536924413 +0000 UTC m=+1348.723517720"
Jul 28 17:39:09.184785 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:09.184740    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.31:8643/healthz\": dial tcp 10.132.0.31:8643: connect: connection refused"
Jul 28 17:39:09.509293 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:09.509260    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:09.509438 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:09.509355    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:09.510909 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:09.510884    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:10.197258 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.197217    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.30:8643/healthz\": dial tcp 10.132.0.30:8643: connect: connection refused"
Jul 28 17:39:10.512825 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.512790    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:10.724938 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.724911    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:39:10.841436 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.841409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:39:10.885969 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.885934    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") pod \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") "
Jul 28 17:39:10.886106 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.885998    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5l4z\" (UniqueName: \"kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z\") pod \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") "
Jul 28 17:39:10.886106 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.886052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config\") pod \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\" (UID: \"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16\") "
Jul 28 17:39:10.886404 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.886379    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" (UID: "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:39:10.888087 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.888064    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" (UID: "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:10.888185 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.888097    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z" (OuterVolumeSpecName: "kube-api-access-z5l4z") pod "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" (UID: "2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16"). InnerVolumeSpecName "kube-api-access-z5l4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:10.986763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.986729    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls\") pod \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") "
Jul 28 17:39:10.986763 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.986767    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config\") pod \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") "
Jul 28 17:39:10.986979 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.986857    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n5mcn\" (UniqueName: \"kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn\") pod \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\" (UID: \"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3\") "
Jul 28 17:39:10.987021 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.987014    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:10.987055 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.987026    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l4z\" (UniqueName: \"kubernetes.io/projected/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-api-access-z5l4z\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:10.987055 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.987039    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:10.987184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.987156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" (UID: "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:39:10.988905 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.988875    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" (UID: "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:39:10.988995 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:10.988925    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn" (OuterVolumeSpecName: "kube-api-access-n5mcn") pod "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" (UID: "b0c82467-5c68-4d9e-bf2f-599d5c8e37d3"). InnerVolumeSpecName "kube-api-access-n5mcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:39:11.087506 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.087421    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5mcn\" (UniqueName: \"kubernetes.io/projected/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-api-access-n5mcn\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:11.087506 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.087453    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:11.087506 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.087465    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:39:11.518036 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.518004    2576 generic.go:356] "Generic (PLEG): container finished" podID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerID="641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb" exitCode=0
Jul 28 17:39:11.518519 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.518060    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"
Jul 28 17:39:11.518519 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.518097    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerDied","Data":"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"}
Jul 28 17:39:11.518519 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.518175    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz" event={"ID":"b0c82467-5c68-4d9e-bf2f-599d5c8e37d3","Type":"ContainerDied","Data":"d937566c0d737d18c3acdc3c68d19286d85a458ea2cffcb4a7055ca70bcb3bba"}
Jul 28 17:39:11.518519 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.518211    2576 scope.go:117] "RemoveContainer" containerID="f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"
Jul 28 17:39:11.519682 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.519661    2576 generic.go:356] "Generic (PLEG): container finished" podID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerID="5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512" exitCode=0
Jul 28 17:39:11.519795 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.519716    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"
Jul 28 17:39:11.519795 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.519750    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerDied","Data":"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"}
Jul 28 17:39:11.519795 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.519789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx" event={"ID":"2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16","Type":"ContainerDied","Data":"f7199a05ea0da2d49675296b4b2ca66ba165d4834242e978f74fb8ce57b15a1e"}
Jul 28 17:39:11.527908 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.527885    2576 scope.go:117] "RemoveContainer" containerID="641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"
Jul 28 17:39:11.537198 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.537172    2576 scope.go:117] "RemoveContainer" containerID="f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"
Jul 28 17:39:11.537492 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:11.537473    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e\": container with ID starting with f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e not found: ID does not exist" containerID="f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"
Jul 28 17:39:11.537659 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.537501    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e"} err="failed to get container status \"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e\": rpc error: code = NotFound desc = could not find container \"f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e\": container with ID starting with f1fa45398559cc5a8ba38418547c00201eb4c128a73a459bd078d3d1d31abf8e not found: ID does not exist"
Jul 28 17:39:11.537659 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.537520    2576 scope.go:117] "RemoveContainer" containerID="641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"
Jul 28 17:39:11.538015 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:11.537809    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb\": container with ID starting with 641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb not found: ID does not exist" containerID="641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"
Jul 28 17:39:11.538015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.537828    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb"} err="failed to get container status \"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb\": rpc error: code = NotFound desc = could not find container \"641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb\": container with ID starting with 641384c3e096ef68ce05bbf1ddb96cdb56d54c92d9ac3e7bdc75065e3fb8adbb not found: ID does not exist"
Jul 28 17:39:11.538015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.537842    2576 scope.go:117] "RemoveContainer" containerID="1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"
Jul 28 17:39:11.541897 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.541878    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:39:11.545937 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.545916    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4108d-predictor-6fdcc84849-ntwgz"]
Jul 28 17:39:11.551566 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.551526    2576 scope.go:117] "RemoveContainer" containerID="5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"
Jul 28 17:39:11.557925 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.557897    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:39:11.559873 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.559842    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4108d-predictor-7bb6fdb996-q7wdx"]
Jul 28 17:39:11.560228 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.560211    2576 scope.go:117] "RemoveContainer" containerID="1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"
Jul 28 17:39:11.560530 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:11.560510    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5\": container with ID starting with 1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5 not found: ID does not exist" containerID="1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"
Jul 28 17:39:11.560596 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.560539    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5"} err="failed to get container status \"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5\": rpc error: code = NotFound desc = could not find container \"1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5\": container with ID starting with 1bb208b84242a35775d3f62b708833a64ddecfa28ab2f4805b7fbab5c73712e5 not found: ID does not exist"
Jul 28 17:39:11.560596 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.560572    2576 scope.go:117] "RemoveContainer" containerID="5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"
Jul 28 17:39:11.560814 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:39:11.560798    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512\": container with ID starting with 5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512 not found: ID does not exist" containerID="5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"
Jul 28 17:39:11.560852 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:11.560817    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512"} err="failed to get container status \"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512\": rpc error: code = NotFound desc = could not find container \"5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512\": container with ID starting with 5a7d473fe0e2244df93485f2bc1bc82e991d3ebfd6bbb2d5e74db8f8beeab512 not found: ID does not exist"
Jul 28 17:39:12.358288 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:12.358255    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" path="/var/lib/kubelet/pods/2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16/volumes"
Jul 28 17:39:12.358745 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:12.358731    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" path="/var/lib/kubelet/pods/b0c82467-5c68-4d9e-bf2f-599d5c8e37d3/volumes"
Jul 28 17:39:14.513409 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:14.513380    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:14.513810 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:14.513773    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:15.517522 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:15.517492    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:39:15.518165 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:15.518137    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:24.513891 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:24.513849    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:25.518203 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:25.518156    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:34.514612 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:34.514569    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:35.518510 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:35.518466    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:44.514762 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:44.514722    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:39:45.519014 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:45.518969    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 28 17:39:54.514660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:54.514623    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:39:55.519740 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:39:55.519704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:47:35.947929 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:35.947891    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:47:35.948352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:35.948167    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" containerID="cri-o://797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4" gracePeriod=30
Jul 28 17:47:35.948352 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:35.948213    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kube-rbac-proxy" containerID="cri-o://9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07" gracePeriod=30
Jul 28 17:47:36.075903 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.075868    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:47:36.076117 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.076086    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" containerID="cri-o://8751c1d14792063f56c6cec82aedaaceeb999840d93c4e914f3853eef4d6273c" gracePeriod=30
Jul 28 17:47:36.076268 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.076119    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kube-rbac-proxy" containerID="cri-o://99840d13833df9aafc954eba17468cd984e2af282c3c5eabb108b5785468c46a" gracePeriod=30
Jul 28 17:47:36.078926 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.078903    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:47:36.079366 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079345    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kube-rbac-proxy"
Jul 28 17:47:36.079366 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079366    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kube-rbac-proxy"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079384    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079393    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079403    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079411    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079429    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kube-rbac-proxy"
Jul 28 17:47:36.079560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079437    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kube-rbac-proxy"
Jul 28 17:47:36.079864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079616    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kube-rbac-proxy"
Jul 28 17:47:36.079864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079635    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7ef913-e7c5-4a2c-9b8a-1b2ec72cfe16" containerName="kserve-container"
Jul 28 17:47:36.079864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079648    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kserve-container"
Jul 28 17:47:36.079864 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.079658    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0c82467-5c68-4d9e-bf2f-599d5c8e37d3" containerName="kube-rbac-proxy"
Jul 28 17:47:36.083364 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.083343    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.086004 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.085984    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32bdd-predictor-serving-cert\""
Jul 28 17:47:36.086091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.085991    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-32bdd-kube-rbac-proxy-sar-config\""
Jul 28 17:47:36.091330 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.091311    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:47:36.133442 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.133406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.133617 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.133466    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.133617 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.133522    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fdvc\" (UniqueName: \"kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.184524 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.184494    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:47:36.188101 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.188074    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.190755 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.190734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-32bdd-predictor-serving-cert\""
Jul 28 17:47:36.190993 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.190737    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-32bdd-kube-rbac-proxy-sar-config\""
Jul 28 17:47:36.197189 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.197166    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:47:36.234482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234360    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.234482 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234443    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff6nn\" (UniqueName: \"kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.234753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234486    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.234753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234566    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.234753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.234753 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.234662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fdvc\" (UniqueName: \"kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.235348 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.235320    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.237018 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.236985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.243206 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.243183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fdvc\" (UniqueName: \"kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc\") pod \"success-200-isvc-32bdd-predictor-68f496d98f-tfnht\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") " pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.335291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.335235    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.335291 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.335295    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff6nn\" (UniqueName: \"kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.335566 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.335323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.335566 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:47:36.335408    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-serving-cert: secret "error-404-isvc-32bdd-predictor-serving-cert" not found
Jul 28 17:47:36.335566 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:47:36.335499    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls podName:1c23fd96-bc7e-4393-8a9d-e5466324a746 nodeName:}" failed. No retries permitted until 2026-07-28 17:47:36.835476327 +0000 UTC m=+1857.022069610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls") pod "error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" (UID: "1c23fd96-bc7e-4393-8a9d-e5466324a746") : secret "error-404-isvc-32bdd-predictor-serving-cert" not found
Jul 28 17:47:36.336041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.336024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.343745 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.343725    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff6nn\" (UniqueName: \"kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.399813 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.399775    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:36.452069 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.451618    2576 generic.go:356] "Generic (PLEG): container finished" podID="a377c3f4-672e-4217-9efd-cbd90e225034" containerID="9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07" exitCode=2
Jul 28 17:47:36.452069 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.451749    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerDied","Data":"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"}
Jul 28 17:47:36.454468 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.454391    2576 generic.go:356] "Generic (PLEG): container finished" podID="45c665a0-4abc-4471-905e-b157af3578c5" containerID="99840d13833df9aafc954eba17468cd984e2af282c3c5eabb108b5785468c46a" exitCode=2
Jul 28 17:47:36.454610 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.454471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerDied","Data":"99840d13833df9aafc954eba17468cd984e2af282c3c5eabb108b5785468c46a"}
Jul 28 17:47:36.531172 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.531133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:47:36.534140 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:47:36.534114    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13baaa32_4f7e_42e5_8d7f_e5020b670207.slice/crio-394b4d14e1fc86a7f565ccaac5325f00c4bf1172df5d2d1098e7eba2d8f502ca WatchSource:0}: Error finding container 394b4d14e1fc86a7f565ccaac5325f00c4bf1172df5d2d1098e7eba2d8f502ca: Status 404 returned error can't find the container with id 394b4d14e1fc86a7f565ccaac5325f00c4bf1172df5d2d1098e7eba2d8f502ca
Jul 28 17:47:36.838979 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.838866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:36.841465 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:36.841429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") pod \"error-404-isvc-32bdd-predictor-84cb75477b-gsdp8\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") " pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:37.098892 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.098806    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:37.229992 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.229956    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:47:37.233333 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:47:37.233300    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c23fd96_bc7e_4393_8a9d_e5466324a746.slice/crio-8193714961bcc3cd7f3498856f9818808f1a072286d1e12c656bbe263591bcc9 WatchSource:0}: Error finding container 8193714961bcc3cd7f3498856f9818808f1a072286d1e12c656bbe263591bcc9: Status 404 returned error can't find the container with id 8193714961bcc3cd7f3498856f9818808f1a072286d1e12c656bbe263591bcc9
Jul 28 17:47:37.460422 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.460325    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerStarted","Data":"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"}
Jul 28 17:47:37.460422 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.460396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerStarted","Data":"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"}
Jul 28 17:47:37.460658 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.460425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerStarted","Data":"394b4d14e1fc86a7f565ccaac5325f00c4bf1172df5d2d1098e7eba2d8f502ca"}
Jul 28 17:47:37.460658 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.460450    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:37.462124 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.462103    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerStarted","Data":"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"}
Jul 28 17:47:37.462194 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.462151    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:37.462194 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.462171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerStarted","Data":"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"}
Jul 28 17:47:37.462194 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.462189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerStarted","Data":"8193714961bcc3cd7f3498856f9818808f1a072286d1e12c656bbe263591bcc9"}
Jul 28 17:47:37.462309 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.462208    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:37.463436 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.463413    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:47:37.511657 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.511608    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podStartSLOduration=1.511590588 podStartE2EDuration="1.511590588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:37.485881897 +0000 UTC m=+1857.672475202" watchObservedRunningTime="2026-07-28 17:47:37.511590588 +0000 UTC m=+1857.698183895"
Jul 28 17:47:37.512846 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:37.512811    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podStartSLOduration=1.512798987 podStartE2EDuration="1.512798987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:47:37.511560507 +0000 UTC m=+1857.698153803" watchObservedRunningTime="2026-07-28 17:47:37.512798987 +0000 UTC m=+1857.699392293"
Jul 28 17:47:38.261068 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.261022    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.33:8643/healthz\": dial tcp 10.132.0.33:8643: connect: connection refused"
Jul 28 17:47:38.261459 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.261037    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.32:8643/healthz\": dial tcp 10.132.0.32:8643: connect: connection refused"
Jul 28 17:47:38.266336 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.266306    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.32:8080: connect: connection refused"
Jul 28 17:47:38.266450 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.266314    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.33:8080: connect: connection refused"
Jul 28 17:47:38.467197 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.467154    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:47:38.467394 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.467292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:38.468496 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:38.468466    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:47:39.405487 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.405458    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:47:39.457781 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.457754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mfbsn\" (UniqueName: \"kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn\") pod \"a377c3f4-672e-4217-9efd-cbd90e225034\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") "
Jul 28 17:47:39.457901 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.457796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls\") pod \"a377c3f4-672e-4217-9efd-cbd90e225034\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") "
Jul 28 17:47:39.457901 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.457817    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config\") pod \"a377c3f4-672e-4217-9efd-cbd90e225034\" (UID: \"a377c3f4-672e-4217-9efd-cbd90e225034\") "
Jul 28 17:47:39.458196 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.458171    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a377c3f4-672e-4217-9efd-cbd90e225034" (UID: "a377c3f4-672e-4217-9efd-cbd90e225034"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:39.459748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.459728    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn" (OuterVolumeSpecName: "kube-api-access-mfbsn") pod "a377c3f4-672e-4217-9efd-cbd90e225034" (UID: "a377c3f4-672e-4217-9efd-cbd90e225034"). InnerVolumeSpecName "kube-api-access-mfbsn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:39.459819 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.459790    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a377c3f4-672e-4217-9efd-cbd90e225034" (UID: "a377c3f4-672e-4217-9efd-cbd90e225034"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:39.472670 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.472639    2576 generic.go:356] "Generic (PLEG): container finished" podID="a377c3f4-672e-4217-9efd-cbd90e225034" containerID="797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4" exitCode=0
Jul 28 17:47:39.472781 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.472693    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"
Jul 28 17:47:39.472781 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.472720    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerDied","Data":"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"}
Jul 28 17:47:39.472851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.472788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk" event={"ID":"a377c3f4-672e-4217-9efd-cbd90e225034","Type":"ContainerDied","Data":"86b3efe50fe3cc8b0356b427cd0350dc7f8d2fac9969992f9ba49150f8ed31d4"}
Jul 28 17:47:39.472851 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.472826    2576 scope.go:117] "RemoveContainer" containerID="9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"
Jul 28 17:47:39.473318 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.473291    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:47:39.495061 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.495037    2576 scope.go:117] "RemoveContainer" containerID="797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"
Jul 28 17:47:39.502821 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.502796    2576 scope.go:117] "RemoveContainer" containerID="9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"
Jul 28 17:47:39.503092 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:47:39.503070    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07\": container with ID starting with 9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07 not found: ID does not exist" containerID="9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"
Jul 28 17:47:39.503153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.503107    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07"} err="failed to get container status \"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07\": rpc error: code = NotFound desc = could not find container \"9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07\": container with ID starting with 9a064392f7389c77ac9602d331c5246a0a1a9a6919c9132d48855e3541768f07 not found: ID does not exist"
Jul 28 17:47:39.503153 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.503127    2576 scope.go:117] "RemoveContainer" containerID="797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"
Jul 28 17:47:39.503377 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:47:39.503362    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4\": container with ID starting with 797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4 not found: ID does not exist" containerID="797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"
Jul 28 17:47:39.503425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.503379    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4"} err="failed to get container status \"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4\": rpc error: code = NotFound desc = could not find container \"797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4\": container with ID starting with 797a813947edc972182da3659f6fb359cffc35b7a455122fbea113287099a2e4 not found: ID does not exist"
Jul 28 17:47:39.507332 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.507308    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:47:39.512775 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.512751    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c57a1-predictor-6674cb8b85-xxzrk"]
Jul 28 17:47:39.559459 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.559419    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfbsn\" (UniqueName: \"kubernetes.io/projected/a377c3f4-672e-4217-9efd-cbd90e225034-kube-api-access-mfbsn\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:39.559459 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.559458    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a377c3f4-672e-4217-9efd-cbd90e225034-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:39.559665 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.559473    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a377c3f4-672e-4217-9efd-cbd90e225034-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:39.608977 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.608955    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:47:39.659844 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.659798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls\") pod \"45c665a0-4abc-4471-905e-b157af3578c5\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") "
Jul 28 17:47:39.660015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.659891    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config\") pod \"45c665a0-4abc-4471-905e-b157af3578c5\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") "
Jul 28 17:47:39.660232 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.660199    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "45c665a0-4abc-4471-905e-b157af3578c5" (UID: "45c665a0-4abc-4471-905e-b157af3578c5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:47:39.661750 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.661730    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "45c665a0-4abc-4471-905e-b157af3578c5" (UID: "45c665a0-4abc-4471-905e-b157af3578c5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:47:39.761078 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.761040    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hnqx\" (UniqueName: \"kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx\") pod \"45c665a0-4abc-4471-905e-b157af3578c5\" (UID: \"45c665a0-4abc-4471-905e-b157af3578c5\") "
Jul 28 17:47:39.761241 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.761182    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/45c665a0-4abc-4471-905e-b157af3578c5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:39.761241 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.761194    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/45c665a0-4abc-4471-905e-b157af3578c5-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:39.763029 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.763006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx" (OuterVolumeSpecName: "kube-api-access-8hnqx") pod "45c665a0-4abc-4471-905e-b157af3578c5" (UID: "45c665a0-4abc-4471-905e-b157af3578c5"). InnerVolumeSpecName "kube-api-access-8hnqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:47:39.862200 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:39.862155    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hnqx\" (UniqueName: \"kubernetes.io/projected/45c665a0-4abc-4471-905e-b157af3578c5-kube-api-access-8hnqx\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:47:40.358400 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.358367    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" path="/var/lib/kubelet/pods/a377c3f4-672e-4217-9efd-cbd90e225034/volumes"
Jul 28 17:47:40.432632 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.432606    2576 scope.go:117] "RemoveContainer" containerID="8751c1d14792063f56c6cec82aedaaceeb999840d93c4e914f3853eef4d6273c"
Jul 28 17:47:40.443265 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.443235    2576 scope.go:117] "RemoveContainer" containerID="99840d13833df9aafc954eba17468cd984e2af282c3c5eabb108b5785468c46a"
Jul 28 17:47:40.476076 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.476048    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerDied","Data":"8751c1d14792063f56c6cec82aedaaceeb999840d93c4e914f3853eef4d6273c"}
Jul 28 17:47:40.476189 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.476125    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s" event={"ID":"45c665a0-4abc-4471-905e-b157af3578c5","Type":"ContainerDied","Data":"c1ffb9f1eda23a30a42fe3ffb37e142a69f76f3f0382b26feb19615bcbb87079"}
Jul 28 17:47:40.476189 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.476060    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"
Jul 28 17:47:40.510660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.510620    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:47:40.512908 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:40.512881    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c57a1-predictor-9bd97b66d-twn7s"]
Jul 28 17:47:42.358140 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:42.358106    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c665a0-4abc-4471-905e-b157af3578c5" path="/var/lib/kubelet/pods/45c665a0-4abc-4471-905e-b157af3578c5/volumes"
Jul 28 17:47:43.471676 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:43.471640    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:47:43.472078 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:43.472055    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:47:44.477603 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:44.477575    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:47:44.477986 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:44.477944    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:47:53.472785 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:53.472743    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:47:54.478090 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:47:54.478051    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:48:03.473074 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:03.472981    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:48:04.478271 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:04.478233    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:48:13.472522 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:13.472479    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.37:8080: connect: connection refused"
Jul 28 17:48:14.478845 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:14.478806    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.36:8080: connect: connection refused"
Jul 28 17:48:21.782982 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.782934    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:48:21.783380 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.783227    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container" containerID="cri-o://7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b" gracePeriod=30
Jul 28 17:48:21.783380 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.783258    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kube-rbac-proxy" containerID="cri-o://c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3" gracePeriod=30
Jul 28 17:48:21.835693 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.835657    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:48:21.836278 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.836244    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" containerID="cri-o://a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660" gracePeriod=30
Jul 28 17:48:21.837151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.836612    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kube-rbac-proxy" containerID="cri-o://1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08" gracePeriod=30
Jul 28 17:48:21.862835 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.862801    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 17:48:21.863172 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863159    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863173    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863183    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863189    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863197    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863202    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863213    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863220 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863218    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863269    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863276    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kserve-container"
Jul 28 17:48:21.863425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863283    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="45c665a0-4abc-4471-905e-b157af3578c5" containerName="kube-rbac-proxy"
Jul 28 17:48:21.863425 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.863289    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a377c3f4-672e-4217-9efd-cbd90e225034" containerName="kserve-container"
Jul 28 17:48:21.866353 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.866332    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:21.868813 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.868785    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c0ed1-predictor-serving-cert\""
Jul 28 17:48:21.868910 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.868790    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c0ed1-kube-rbac-proxy-sar-config\""
Jul 28 17:48:21.876615 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.876589    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 17:48:21.962784 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.962755    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 17:48:21.966195 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.966173    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:21.969221 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.969196    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c0ed1-predictor-serving-cert\""
Jul 28 17:48:21.969673 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.969652    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c0ed1-kube-rbac-proxy-sar-config\""
Jul 28 17:48:21.977865 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:21.977843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 17:48:22.029130 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.029099    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.029294 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.029159    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.029294 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.029243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kz9c\" (UniqueName: \"kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.129818 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.129818 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrdw\" (UniqueName: \"kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.129818 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.129818 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129824    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.130155 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129843    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kz9c\" (UniqueName: \"kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.130155 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.129886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.130604 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.130578    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.132519 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.132490    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.139201 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.139173    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kz9c\" (UniqueName: \"kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c\") pod \"success-200-isvc-c0ed1-predictor-76c894557-m64nt\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") " pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.177146 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.177109    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.230364    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.230421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrdw\" (UniqueName: \"kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.230470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:22.231152    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-serving-cert: secret "error-404-isvc-c0ed1-predictor-serving-cert" not found
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:22.231234    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls podName:7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a nodeName:}" failed. No retries permitted until 2026-07-28 17:48:22.731214744 +0000 UTC m=+1902.917808031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls") pod "error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" (UID: "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a") : secret "error-404-isvc-c0ed1-predictor-serving-cert" not found
Jul 28 17:48:22.231341 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.231255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.240708 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.240676    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrdw\" (UniqueName: \"kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.312751 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.312574    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 17:48:22.314969 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:48:22.314936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ad12a3_5566_454f_9129_57fa4b3e5495.slice/crio-a4ccf44a15d0fb516ab37c98e810ab5211a6b8a9afbf6c68ae8093015d110068 WatchSource:0}: Error finding container a4ccf44a15d0fb516ab37c98e810ab5211a6b8a9afbf6c68ae8093015d110068: Status 404 returned error can't find the container with id a4ccf44a15d0fb516ab37c98e810ab5211a6b8a9afbf6c68ae8093015d110068
Jul 28 17:48:22.644083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.643987    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerStarted","Data":"0bc191f372bf77a3de5d3216db854c09010f8a549c707d800b096433c4d37b9e"}
Jul 28 17:48:22.644083 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.644062    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerStarted","Data":"2c879bbd640dded0a84961eb739ddf2cb7de2324b79e5716e3102d9698552d2c"}
Jul 28 17:48:22.644316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.644094    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerStarted","Data":"a4ccf44a15d0fb516ab37c98e810ab5211a6b8a9afbf6c68ae8093015d110068"}
Jul 28 17:48:22.644316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.644118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.644316 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.644258    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:22.645890 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.645860    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:22.646015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.645937    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerID="c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3" exitCode=2
Jul 28 17:48:22.646015 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.646003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerDied","Data":"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"}
Jul 28 17:48:22.647608 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.647586    2576 generic.go:356] "Generic (PLEG): container finished" podID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerID="1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08" exitCode=2
Jul 28 17:48:22.647707 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.647620    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerDied","Data":"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"}
Jul 28 17:48:22.663447 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.663401    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podStartSLOduration=1.663383689 podStartE2EDuration="1.663383689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:22.661379363 +0000 UTC m=+1902.847972667" watchObservedRunningTime="2026-07-28 17:48:22.663383689 +0000 UTC m=+1902.849977062"
Jul 28 17:48:22.735508 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.735466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.738225 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.738191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") pod \"error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") " pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:22.879412 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:22.879368    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:23.032302 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.032246    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 17:48:23.034815 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:48:23.034790    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df2b1d2_0e0c_47a2_b152_e21b00bc8e1a.slice/crio-f3d599eeea6a29776c23c72fc997f67ae7ce9ba4c6c1bf61c33d3cffbc931e41 WatchSource:0}: Error finding container f3d599eeea6a29776c23c72fc997f67ae7ce9ba4c6c1bf61c33d3cffbc931e41: Status 404 returned error can't find the container with id f3d599eeea6a29776c23c72fc997f67ae7ce9ba4c6c1bf61c33d3cffbc931e41
Jul 28 17:48:23.473349 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.473317    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:48:23.652748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.652716    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerStarted","Data":"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"}
Jul 28 17:48:23.652928 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.652776    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerStarted","Data":"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"}
Jul 28 17:48:23.652928 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.652812    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:23.652928 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.652865    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerStarted","Data":"f3d599eeea6a29776c23c72fc997f67ae7ce9ba4c6c1bf61c33d3cffbc931e41"}
Jul 28 17:48:23.653092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.653031    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:23.653092 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.653063    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:23.654301 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.654275    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:48:23.672248 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:23.672201    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podStartSLOduration=2.672187954 podStartE2EDuration="2.672187954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:23.670210818 +0000 UTC m=+1903.856804122" watchObservedRunningTime="2026-07-28 17:48:23.672187954 +0000 UTC m=+1903.858781258"
Jul 28 17:48:24.479741 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:24.479707    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:48:24.510486 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:24.510444    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.35:8643/healthz\": dial tcp 10.132.0.35:8643: connect: connection refused"
Jul 28 17:48:24.513872 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:24.513836    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.35:8080: connect: connection refused"
Jul 28 17:48:24.657187 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:24.657151    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:48:25.309724 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.309701    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:48:25.445318 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.445294    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:48:25.459173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.459139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config\") pod \"d8a795a5-5625-4829-8306-0fc96a2377dd\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") "
Jul 28 17:48:25.459173 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.459172    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qql7\" (UniqueName: \"kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7\") pod \"d8a795a5-5625-4829-8306-0fc96a2377dd\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") "
Jul 28 17:48:25.459388 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.459228    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") pod \"d8a795a5-5625-4829-8306-0fc96a2377dd\" (UID: \"d8a795a5-5625-4829-8306-0fc96a2377dd\") "
Jul 28 17:48:25.459811 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.459486    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d8a795a5-5625-4829-8306-0fc96a2377dd" (UID: "d8a795a5-5625-4829-8306-0fc96a2377dd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:25.461422 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.461400    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d8a795a5-5625-4829-8306-0fc96a2377dd" (UID: "d8a795a5-5625-4829-8306-0fc96a2377dd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:25.461518 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.461470    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7" (OuterVolumeSpecName: "kube-api-access-5qql7") pod "d8a795a5-5625-4829-8306-0fc96a2377dd" (UID: "d8a795a5-5625-4829-8306-0fc96a2377dd"). InnerVolumeSpecName "kube-api-access-5qql7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:25.559793 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.559759    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config\") pod \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") "
Jul 28 17:48:25.560184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.559854    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckrk7\" (UniqueName: \"kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7\") pod \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") "
Jul 28 17:48:25.560184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.559884    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls\") pod \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\" (UID: \"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1\") "
Jul 28 17:48:25.560184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.560017    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.560184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.560033    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qql7\" (UniqueName: \"kubernetes.io/projected/d8a795a5-5625-4829-8306-0fc96a2377dd-kube-api-access-5qql7\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.560184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.560047    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d8a795a5-5625-4829-8306-0fc96a2377dd-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.560356 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.560189    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" (UID: "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:25.561929 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.561907    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7" (OuterVolumeSpecName: "kube-api-access-ckrk7") pod "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" (UID: "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1"). InnerVolumeSpecName "kube-api-access-ckrk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:25.561986 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.561928    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" (UID: "a3668a6c-7b0b-4814-9ced-5f29d4f88ad1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:25.660490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.660464    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckrk7\" (UniqueName: \"kubernetes.io/projected/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-api-access-ckrk7\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.660490 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.660488    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.660692 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.660500    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:25.662926 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.662895    2576 generic.go:356] "Generic (PLEG): container finished" podID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerID="7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b" exitCode=0
Jul 28 17:48:25.663049 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.662947    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"
Jul 28 17:48:25.663049 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.662974    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerDied","Data":"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"}
Jul 28 17:48:25.663049 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.663037    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7" event={"ID":"a3668a6c-7b0b-4814-9ced-5f29d4f88ad1","Type":"ContainerDied","Data":"b568d977196833ac35db9fbff4644584001f45b7f5fba80c0af0d639368125df"}
Jul 28 17:48:25.663198 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.663068    2576 scope.go:117] "RemoveContainer" containerID="c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"
Jul 28 17:48:25.664732 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.664683    2576 generic.go:356] "Generic (PLEG): container finished" podID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerID="a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660" exitCode=0
Jul 28 17:48:25.664827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.664733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerDied","Data":"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"}
Jul 28 17:48:25.664827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.664749    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"
Jul 28 17:48:25.664827 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.664796    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt" event={"ID":"d8a795a5-5625-4829-8306-0fc96a2377dd","Type":"ContainerDied","Data":"9c1df0d3d471dbe8a5202ea60b4074b87b290074356a08a42c8e40470a652d36"}
Jul 28 17:48:25.673337 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.673320    2576 scope.go:117] "RemoveContainer" containerID="7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"
Jul 28 17:48:25.681334 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.681315    2576 scope.go:117] "RemoveContainer" containerID="c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"
Jul 28 17:48:25.681607 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:25.681588    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3\": container with ID starting with c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3 not found: ID does not exist" containerID="c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"
Jul 28 17:48:25.681660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.681618    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3"} err="failed to get container status \"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3\": rpc error: code = NotFound desc = could not find container \"c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3\": container with ID starting with c6fed5ad4b5508c74c05dfbdabadd1b0fc8068a20e7ec80897e75eeed9751aa3 not found: ID does not exist"
Jul 28 17:48:25.681660 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.681640    2576 scope.go:117] "RemoveContainer" containerID="7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"
Jul 28 17:48:25.681907 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:25.681889    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b\": container with ID starting with 7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b not found: ID does not exist" containerID="7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"
Jul 28 17:48:25.681965 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.681913    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b"} err="failed to get container status \"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b\": rpc error: code = NotFound desc = could not find container \"7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b\": container with ID starting with 7ddd1135cf1cbbf6aec4ba0683fbb878c1329675b8b4e20f8c9574cb88a2b38b not found: ID does not exist"
Jul 28 17:48:25.681965 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.681928    2576 scope.go:117] "RemoveContainer" containerID="1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"
Jul 28 17:48:25.687079 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.687057    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:48:25.689750 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.689732    2576 scope.go:117] "RemoveContainer" containerID="a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"
Jul 28 17:48:25.691184 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.691166    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c7add-predictor-fb5c8d6b9-gkcg7"]
Jul 28 17:48:25.696999 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.696984    2576 scope.go:117] "RemoveContainer" containerID="1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"
Jul 28 17:48:25.697237 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:25.697219    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08\": container with ID starting with 1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08 not found: ID does not exist" containerID="1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"
Jul 28 17:48:25.697290 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.697243    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08"} err="failed to get container status \"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08\": rpc error: code = NotFound desc = could not find container \"1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08\": container with ID starting with 1c89fda02621e223ce96b4daaa95a4e84d1e06fc183bfe9ce12fcca394c87a08 not found: ID does not exist"
Jul 28 17:48:25.697290 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.697261    2576 scope.go:117] "RemoveContainer" containerID="a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"
Jul 28 17:48:25.697472 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:25.697456    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660\": container with ID starting with a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660 not found: ID does not exist" containerID="a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"
Jul 28 17:48:25.697511 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.697477    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660"} err="failed to get container status \"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660\": rpc error: code = NotFound desc = could not find container \"a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660\": container with ID starting with a883d59ba2eb3b47313aee31ba18e25f74aacff1edc6f1aa879b32ff79481660 not found: ID does not exist"
Jul 28 17:48:25.702444 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.702421    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:48:25.709087 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:25.709067    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c7add-predictor-6d685ddd8-cd8kt"]
Jul 28 17:48:26.357965 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:26.357936    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" path="/var/lib/kubelet/pods/a3668a6c-7b0b-4814-9ced-5f29d4f88ad1/volumes"
Jul 28 17:48:26.358403 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:26.358388    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" path="/var/lib/kubelet/pods/d8a795a5-5625-4829-8306-0fc96a2377dd/volumes"
Jul 28 17:48:28.657157 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:28.657130    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:48:28.657673 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:28.657647    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:29.661595 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:29.661562    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:48:29.662103 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:29.662079    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:48:38.657950 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:38.657912    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:39.662775 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:39.662727    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:48:46.239287 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.239167    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:48:46.239743 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.239456    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container" containerID="cri-o://b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8" gracePeriod=30
Jul 28 17:48:46.239743 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.239501    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kube-rbac-proxy" containerID="cri-o://dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9" gracePeriod=30
Jul 28 17:48:46.295331 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.295298    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:48:46.295581 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.295535    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container" containerID="cri-o://85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0" gracePeriod=30
Jul 28 17:48:46.295688 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.295601    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kube-rbac-proxy" containerID="cri-o://58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be" gracePeriod=30
Jul 28 17:48:46.359087 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359056    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:48:46.359439 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359422    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359441    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359460    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kube-rbac-proxy"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359468    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kube-rbac-proxy"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359482    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kube-rbac-proxy"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359491    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kube-rbac-proxy"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359505    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container"
Jul 28 17:48:46.359528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359514    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container"
Jul 28 17:48:46.359889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359638    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kube-rbac-proxy"
Jul 28 17:48:46.359889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359650    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3668a6c-7b0b-4814-9ced-5f29d4f88ad1" containerName="kserve-container"
Jul 28 17:48:46.359889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359663    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kserve-container"
Jul 28 17:48:46.359889 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.359671    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8a795a5-5625-4829-8306-0fc96a2377dd" containerName="kube-rbac-proxy"
Jul 28 17:48:46.362908 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.362887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.365668 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.365640    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80e89-kube-rbac-proxy-sar-config\""
Jul 28 17:48:46.365813 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.365710    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-80e89-predictor-serving-cert\""
Jul 28 17:48:46.367489 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.367460    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:48:46.440527 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.440488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.440716 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.440573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.440716 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.440661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.475041 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.474999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:48:46.478307 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.478287    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.481011 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.480988    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80e89-predictor-serving-cert\""
Jul 28 17:48:46.481124 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.481089    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-80e89-kube-rbac-proxy-sar-config\""
Jul 28 17:48:46.488228 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.488203    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:48:46.541576 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541515    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.541774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.541774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.541774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541676    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.541774 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541716    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddxvk\" (UniqueName: \"kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.542002 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.541898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.542471 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.542447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.544212 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.544191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.551879 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.551853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8\") pod \"success-200-isvc-80e89-predictor-7b866dd6f9-smth2\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") " pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.642932 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.642895    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.643135 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.642980    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.643135 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.643023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddxvk\" (UniqueName: \"kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.643135 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:46.643056    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-80e89-predictor-serving-cert: secret "error-404-isvc-80e89-predictor-serving-cert" not found
Jul 28 17:48:46.643135 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:46.643124    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls podName:23af64eb-22ef-421f-8f81-536bada20b4e nodeName:}" failed. No retries permitted until 2026-07-28 17:48:47.14310687 +0000 UTC m=+1927.329700152 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls") pod "error-404-isvc-80e89-predictor-f86d55dd6-56mdh" (UID: "23af64eb-22ef-421f-8f81-536bada20b4e") : secret "error-404-isvc-80e89-predictor-serving-cert" not found
Jul 28 17:48:46.643672 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.643653    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.656112 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.656093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddxvk\" (UniqueName: \"kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:46.676523 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.676494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:46.750702 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.750671    2576 generic.go:356] "Generic (PLEG): container finished" podID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerID="58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be" exitCode=2
Jul 28 17:48:46.750856 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.750740    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerDied","Data":"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"}
Jul 28 17:48:46.752445 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.752421    2576 generic.go:356] "Generic (PLEG): container finished" podID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerID="dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9" exitCode=2
Jul 28 17:48:46.752586 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.752481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerDied","Data":"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"}
Jul 28 17:48:46.818364 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:46.818282    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:48:46.821128 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:48:46.821099    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8be5a8_0cef_4fb8_9289_c32719c96da1.slice/crio-6dd1126d3308639de8d8051d6039a34ce3d9a08e4993a593f3a81b24d2cbf7d4 WatchSource:0}: Error finding container 6dd1126d3308639de8d8051d6039a34ce3d9a08e4993a593f3a81b24d2cbf7d4: Status 404 returned error can't find the container with id 6dd1126d3308639de8d8051d6039a34ce3d9a08e4993a593f3a81b24d2cbf7d4
Jul 28 17:48:47.147680 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.147599    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:47.149959 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.149938    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") pod \"error-404-isvc-80e89-predictor-f86d55dd6-56mdh\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") " pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:47.389235 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.389191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:47.527678 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.527653    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:48:47.530227 ip-10-0-134-28 kubenswrapper[2576]: W0728 17:48:47.530192    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23af64eb_22ef_421f_8f81_536bada20b4e.slice/crio-95e404f99058e4a5cce540b7d426ec5f17059e1d49c3e226aacac3c45df6f0ae WatchSource:0}: Error finding container 95e404f99058e4a5cce540b7d426ec5f17059e1d49c3e226aacac3c45df6f0ae: Status 404 returned error can't find the container with id 95e404f99058e4a5cce540b7d426ec5f17059e1d49c3e226aacac3c45df6f0ae
Jul 28 17:48:47.764249 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.764211    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerStarted","Data":"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"}
Jul 28 17:48:47.764429 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.764311    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerStarted","Data":"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"}
Jul 28 17:48:47.764429 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.764348    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:47.764429 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.764376    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:47.764429 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.764401    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerStarted","Data":"6dd1126d3308639de8d8051d6039a34ce3d9a08e4993a593f3a81b24d2cbf7d4"}
Jul 28 17:48:47.766105 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766076    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerStarted","Data":"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"}
Jul 28 17:48:47.766252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766135    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerStarted","Data":"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"}
Jul 28 17:48:47.766252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766132    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:48:47.766252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766169    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerStarted","Data":"95e404f99058e4a5cce540b7d426ec5f17059e1d49c3e226aacac3c45df6f0ae"}
Jul 28 17:48:47.766252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766202    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:47.766252 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.766230    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:47.767259 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.767239    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:48:47.783336 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:47.783295    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podStartSLOduration=1.783280243 podStartE2EDuration="1.783280243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:47.7823938 +0000 UTC m=+1927.968987105" watchObservedRunningTime="2026-07-28 17:48:47.783280243 +0000 UTC m=+1927.969873548"
Jul 28 17:48:48.467765 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:48.467726    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.37:8643/healthz\": dial tcp 10.132.0.37:8643: connect: connection refused"
Jul 28 17:48:48.658513 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:48.658472    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:48.770748 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:48.770705    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:48:48.770940 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:48.770806    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:48:49.473983 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.473945    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.36:8643/healthz\": dial tcp 10.132.0.36:8643: connect: connection refused"
Jul 28 17:48:49.592531 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.592505    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:48:49.614091 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.614045    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podStartSLOduration=3.614028496 podStartE2EDuration="3.614028496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:47.810272078 +0000 UTC m=+1927.996865383" watchObservedRunningTime="2026-07-28 17:48:49.614028496 +0000 UTC m=+1929.800621801"
Jul 28 17:48:49.662467 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.662430    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:48:49.670681 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.670657    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") pod \"1c23fd96-bc7e-4393-8a9d-e5466324a746\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") "
Jul 28 17:48:49.670797 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.670731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ff6nn\" (UniqueName: \"kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn\") pod \"1c23fd96-bc7e-4393-8a9d-e5466324a746\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") "
Jul 28 17:48:49.670899 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.670876    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config\") pod \"1c23fd96-bc7e-4393-8a9d-e5466324a746\" (UID: \"1c23fd96-bc7e-4393-8a9d-e5466324a746\") "
Jul 28 17:48:49.671189 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.671163    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c23fd96-bc7e-4393-8a9d-e5466324a746" (UID: "1c23fd96-bc7e-4393-8a9d-e5466324a746"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:49.672775 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.672749    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c23fd96-bc7e-4393-8a9d-e5466324a746" (UID: "1c23fd96-bc7e-4393-8a9d-e5466324a746"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:49.672853 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.672752    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn" (OuterVolumeSpecName: "kube-api-access-ff6nn") pod "1c23fd96-bc7e-4393-8a9d-e5466324a746" (UID: "1c23fd96-bc7e-4393-8a9d-e5466324a746"). InnerVolumeSpecName "kube-api-access-ff6nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:49.771518 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.771488    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff6nn\" (UniqueName: \"kubernetes.io/projected/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-api-access-ff6nn\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:49.771518 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.771516    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c23fd96-bc7e-4393-8a9d-e5466324a746-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:49.771727 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.771528    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c23fd96-bc7e-4393-8a9d-e5466324a746-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:49.776031 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.776005    2576 generic.go:356] "Generic (PLEG): container finished" podID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerID="b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8" exitCode=0
Jul 28 17:48:49.776142 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.776058    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"
Jul 28 17:48:49.776142 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.776079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerDied","Data":"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"}
Jul 28 17:48:49.776222 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.776143    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8" event={"ID":"1c23fd96-bc7e-4393-8a9d-e5466324a746","Type":"ContainerDied","Data":"8193714961bcc3cd7f3498856f9818808f1a072286d1e12c656bbe263591bcc9"}
Jul 28 17:48:49.776222 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.776169    2576 scope.go:117] "RemoveContainer" containerID="dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"
Jul 28 17:48:49.805624 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.805593    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:48:49.806577 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.806558    2576 scope.go:117] "RemoveContainer" containerID="b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"
Jul 28 17:48:49.814735 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.814714    2576 scope.go:117] "RemoveContainer" containerID="dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"
Jul 28 17:48:49.815039 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:49.815011    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9\": container with ID starting with dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9 not found: ID does not exist" containerID="dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"
Jul 28 17:48:49.815109 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.815039    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9"} err="failed to get container status \"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9\": rpc error: code = NotFound desc = could not find container \"dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9\": container with ID starting with dedd5d1a9e5b5457b53079e77b3013c3e234b752bb562020893257e6735f81c9 not found: ID does not exist"
Jul 28 17:48:49.815109 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.815059    2576 scope.go:117] "RemoveContainer" containerID="b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"
Jul 28 17:48:49.815273 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:49.815255    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8\": container with ID starting with b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8 not found: ID does not exist" containerID="b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"
Jul 28 17:48:49.815313 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.815280    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8"} err="failed to get container status \"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8\": rpc error: code = NotFound desc = could not find container \"b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8\": container with ID starting with b6d2ac047185f96fac9344116b1be9cd4d5f8070e6bf1a34bfc22abf882a99f8 not found: ID does not exist"
Jul 28 17:48:49.815607 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.815583    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-32bdd-predictor-84cb75477b-gsdp8"]
Jul 28 17:48:49.928629 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.928605    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:48:49.973025 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.972995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls\") pod \"13baaa32-4f7e-42e5-8d7f-e5020b670207\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") "
Jul 28 17:48:49.973200 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.973041    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config\") pod \"13baaa32-4f7e-42e5-8d7f-e5020b670207\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") "
Jul 28 17:48:49.973200 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.973064    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fdvc\" (UniqueName: \"kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc\") pod \"13baaa32-4f7e-42e5-8d7f-e5020b670207\" (UID: \"13baaa32-4f7e-42e5-8d7f-e5020b670207\") "
Jul 28 17:48:49.973411 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.973387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13baaa32-4f7e-42e5-8d7f-e5020b670207" (UID: "13baaa32-4f7e-42e5-8d7f-e5020b670207"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:49.975169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.975145    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13baaa32-4f7e-42e5-8d7f-e5020b670207" (UID: "13baaa32-4f7e-42e5-8d7f-e5020b670207"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:49.975169 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:49.975158    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc" (OuterVolumeSpecName: "kube-api-access-8fdvc") pod "13baaa32-4f7e-42e5-8d7f-e5020b670207" (UID: "13baaa32-4f7e-42e5-8d7f-e5020b670207"). InnerVolumeSpecName "kube-api-access-8fdvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:50.073674 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.073630    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13baaa32-4f7e-42e5-8d7f-e5020b670207-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:50.073674 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.073669    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:50.073674 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.073679    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fdvc\" (UniqueName: \"kubernetes.io/projected/13baaa32-4f7e-42e5-8d7f-e5020b670207-kube-api-access-8fdvc\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:48:50.358911 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.358819    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" path="/var/lib/kubelet/pods/1c23fd96-bc7e-4393-8a9d-e5466324a746/volumes"
Jul 28 17:48:50.782287 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.782249    2576 generic.go:356] "Generic (PLEG): container finished" podID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerID="85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0" exitCode=0
Jul 28 17:48:50.782790 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.782301    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"
Jul 28 17:48:50.782790 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.782329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerDied","Data":"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"}
Jul 28 17:48:50.782790 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.782383    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht" event={"ID":"13baaa32-4f7e-42e5-8d7f-e5020b670207","Type":"ContainerDied","Data":"394b4d14e1fc86a7f565ccaac5325f00c4bf1172df5d2d1098e7eba2d8f502ca"}
Jul 28 17:48:50.782790 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.782418    2576 scope.go:117] "RemoveContainer" containerID="58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"
Jul 28 17:48:50.794735 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.791678    2576 scope.go:117] "RemoveContainer" containerID="85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"
Jul 28 17:48:50.800370 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.800350    2576 scope.go:117] "RemoveContainer" containerID="58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"
Jul 28 17:48:50.800691 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:50.800668    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be\": container with ID starting with 58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be not found: ID does not exist" containerID="58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"
Jul 28 17:48:50.800737 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.800700    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be"} err="failed to get container status \"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be\": rpc error: code = NotFound desc = could not find container \"58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be\": container with ID starting with 58a403b698e08e7cfed506797d650083e38a5c5a0fdaf81be9d101c4aa2972be not found: ID does not exist"
Jul 28 17:48:50.800737 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.800717    2576 scope.go:117] "RemoveContainer" containerID="85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"
Jul 28 17:48:50.800978 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:48:50.800960    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0\": container with ID starting with 85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0 not found: ID does not exist" containerID="85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"
Jul 28 17:48:50.801022 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.800983    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0"} err="failed to get container status \"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0\": rpc error: code = NotFound desc = could not find container \"85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0\": container with ID starting with 85a22becb17cb797ef9087cca151021f4f18d0c540e44ffb2ad492d5ab10f9e0 not found: ID does not exist"
Jul 28 17:48:50.806738 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.806715    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:48:50.812501 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:50.812477    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-32bdd-predictor-68f496d98f-tfnht"]
Jul 28 17:48:52.358320 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:52.358284    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" path="/var/lib/kubelet/pods/13baaa32-4f7e-42e5-8d7f-e5020b670207/volumes"
Jul 28 17:48:53.775360 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:53.775326    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:48:53.775824 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:53.775799    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:48:53.775903 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:53.775841    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:48:53.776251 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:53.776224    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:48:58.657953 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:58.657909    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:48:59.662396 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:48:59.662356    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:49:03.775828 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:03.775783    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:49:03.776279 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:03.776248    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:49:08.658262 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:08.658218    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.38:8080: connect: connection refused"
Jul 28 17:49:09.662036 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:09.661995    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.39:8080: connect: connection refused"
Jul 28 17:49:13.776119 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:13.776076    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:49:13.776560 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:13.776220    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:49:18.658454 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:18.658420    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 17:49:19.663303 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:19.663274    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 17:49:23.776150 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:23.776109    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:49:23.776528 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:23.776194    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:49:33.776048 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:33.775950    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:49:33.776460 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:33.776190    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:49:43.776746 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:43.776704    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:49:43.777246 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:49:43.777100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:58:11.235255 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.235214    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:58:11.235837 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.235494    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" containerID="cri-o://ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8" gracePeriod=30
Jul 28 17:58:11.235837 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.235526    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kube-rbac-proxy" containerID="cri-o://06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68" gracePeriod=30
Jul 28 17:58:11.283722 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.283692    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:58:11.283938 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.283914    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" containerID="cri-o://99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276" gracePeriod=30
Jul 28 17:58:11.284005 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.283939    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kube-rbac-proxy" containerID="cri-o://036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624" gracePeriod=30
Jul 28 17:58:11.911920 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.911888    2576 generic.go:356] "Generic (PLEG): container finished" podID="23af64eb-22ef-421f-8f81-536bada20b4e" containerID="036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624" exitCode=2
Jul 28 17:58:11.912099 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.911955    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerDied","Data":"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"}
Jul 28 17:58:11.913711 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.913690    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerID="06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68" exitCode=2
Jul 28 17:58:11.913802 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:11.913727    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerDied","Data":"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"}
Jul 28 17:58:13.771192 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:13.771152    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.40:8643/healthz\": dial tcp 10.132.0.40:8643: connect: connection refused"
Jul 28 17:58:13.771609 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:13.771152    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.41:8643/healthz\": dial tcp 10.132.0.41:8643: connect: connection refused"
Jul 28 17:58:13.775898 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:13.775871    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.40:8080: connect: connection refused"
Jul 28 17:58:13.777202 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:13.777178    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.41:8080: connect: connection refused"
Jul 28 17:58:14.290283 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.290259    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:58:14.457948 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.457866    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls\") pod \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") "
Jul 28 17:58:14.457948 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.457933    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config\") pod \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") "
Jul 28 17:58:14.458151 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.457962    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8\") pod \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\" (UID: \"2c8be5a8-0cef-4fb8-9289-c32719c96da1\") "
Jul 28 17:58:14.458371 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.458343    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c8be5a8-0cef-4fb8-9289-c32719c96da1" (UID: "2c8be5a8-0cef-4fb8-9289-c32719c96da1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:14.460474 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.460451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c8be5a8-0cef-4fb8-9289-c32719c96da1" (UID: "2c8be5a8-0cef-4fb8-9289-c32719c96da1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:14.460584 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.460482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8" (OuterVolumeSpecName: "kube-api-access-66wr8") pod "2c8be5a8-0cef-4fb8-9289-c32719c96da1" (UID: "2c8be5a8-0cef-4fb8-9289-c32719c96da1"). InnerVolumeSpecName "kube-api-access-66wr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:14.559684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.559642    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c8be5a8-0cef-4fb8-9289-c32719c96da1-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:14.559684 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.559680    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:14.559880 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.559694    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66wr8\" (UniqueName: \"kubernetes.io/projected/2c8be5a8-0cef-4fb8-9289-c32719c96da1-kube-api-access-66wr8\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:14.925638 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.925603    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerID="ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8" exitCode=0
Jul 28 17:58:14.926077 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.925650    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"
Jul 28 17:58:14.926077 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.925695    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerDied","Data":"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"}
Jul 28 17:58:14.926077 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.925756    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2" event={"ID":"2c8be5a8-0cef-4fb8-9289-c32719c96da1","Type":"ContainerDied","Data":"6dd1126d3308639de8d8051d6039a34ce3d9a08e4993a593f3a81b24d2cbf7d4"}
Jul 28 17:58:14.926077 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.925784    2576 scope.go:117] "RemoveContainer" containerID="06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"
Jul 28 17:58:14.934811 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.934786    2576 scope.go:117] "RemoveContainer" containerID="ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"
Jul 28 17:58:14.942076 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.942058    2576 scope.go:117] "RemoveContainer" containerID="06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"
Jul 28 17:58:14.942341 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:58:14.942323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68\": container with ID starting with 06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68 not found: ID does not exist" containerID="06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"
Jul 28 17:58:14.942389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.942350    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68"} err="failed to get container status \"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68\": rpc error: code = NotFound desc = could not find container \"06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68\": container with ID starting with 06ca07b79d33ae09d0fc39c2c827c8f4e70e36fd15bd8feefeee7a1de61a8f68 not found: ID does not exist"
Jul 28 17:58:14.942389 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.942366    2576 scope.go:117] "RemoveContainer" containerID="ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"
Jul 28 17:58:14.942613 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:58:14.942596    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8\": container with ID starting with ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8 not found: ID does not exist" containerID="ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"
Jul 28 17:58:14.942668 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.942619    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8"} err="failed to get container status \"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8\": rpc error: code = NotFound desc = could not find container \"ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8\": container with ID starting with ca10704a4c27abc502d0f6184e449685d2854e8960082eb7b6b5b38326cb84e8 not found: ID does not exist"
Jul 28 17:58:14.951295 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.951273    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:58:14.956060 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:14.956035    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-80e89-predictor-7b866dd6f9-smth2"]
Jul 28 17:58:16.358088 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:16.358055    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" path="/var/lib/kubelet/pods/2c8be5a8-0cef-4fb8-9289-c32719c96da1/volumes"
Jul 28 17:58:17.523601 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.523578    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:58:17.685113 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.685024    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddxvk\" (UniqueName: \"kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk\") pod \"23af64eb-22ef-421f-8f81-536bada20b4e\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") "
Jul 28 17:58:17.685113 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.685099    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") pod \"23af64eb-22ef-421f-8f81-536bada20b4e\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") "
Jul 28 17:58:17.685302 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.685124    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config\") pod \"23af64eb-22ef-421f-8f81-536bada20b4e\" (UID: \"23af64eb-22ef-421f-8f81-536bada20b4e\") "
Jul 28 17:58:17.685526 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.685498    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23af64eb-22ef-421f-8f81-536bada20b4e" (UID: "23af64eb-22ef-421f-8f81-536bada20b4e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:58:17.687058 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.687038    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23af64eb-22ef-421f-8f81-536bada20b4e" (UID: "23af64eb-22ef-421f-8f81-536bada20b4e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:58:17.687144 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.687091    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk" (OuterVolumeSpecName: "kube-api-access-ddxvk") pod "23af64eb-22ef-421f-8f81-536bada20b4e" (UID: "23af64eb-22ef-421f-8f81-536bada20b4e"). InnerVolumeSpecName "kube-api-access-ddxvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:58:17.786033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.786002    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddxvk\" (UniqueName: \"kubernetes.io/projected/23af64eb-22ef-421f-8f81-536bada20b4e-kube-api-access-ddxvk\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:17.786033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.786028    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23af64eb-22ef-421f-8f81-536bada20b4e-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:17.786033 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.786039    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23af64eb-22ef-421f-8f81-536bada20b4e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 17:58:17.940606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.940490    2576 generic.go:356] "Generic (PLEG): container finished" podID="23af64eb-22ef-421f-8f81-536bada20b4e" containerID="99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276" exitCode=0
Jul 28 17:58:17.940606 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.940561    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerDied","Data":"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"}
Jul 28 17:58:17.940840 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.940582    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"
Jul 28 17:58:17.940840 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.940675    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh" event={"ID":"23af64eb-22ef-421f-8f81-536bada20b4e","Type":"ContainerDied","Data":"95e404f99058e4a5cce540b7d426ec5f17059e1d49c3e226aacac3c45df6f0ae"}
Jul 28 17:58:17.940840 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.940714    2576 scope.go:117] "RemoveContainer" containerID="036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"
Jul 28 17:58:17.949690 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.949673    2576 scope.go:117] "RemoveContainer" containerID="99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"
Jul 28 17:58:17.956474 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.956454    2576 scope.go:117] "RemoveContainer" containerID="036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"
Jul 28 17:58:17.956745 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:58:17.956721    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624\": container with ID starting with 036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624 not found: ID does not exist" containerID="036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"
Jul 28 17:58:17.956830 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.956751    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624"} err="failed to get container status \"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624\": rpc error: code = NotFound desc = could not find container \"036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624\": container with ID starting with 036b83094e49c69af2a64209d502cfd1c55ce8b8a8d4777bd34f8ee8ea32e624 not found: ID does not exist"
Jul 28 17:58:17.956830 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.956768    2576 scope.go:117] "RemoveContainer" containerID="99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"
Jul 28 17:58:17.956997 ip-10-0-134-28 kubenswrapper[2576]: E0728 17:58:17.956979    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276\": container with ID starting with 99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276 not found: ID does not exist" containerID="99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"
Jul 28 17:58:17.957058 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.957006    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276"} err="failed to get container status \"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276\": rpc error: code = NotFound desc = could not find container \"99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276\": container with ID starting with 99bb34406b118c297d48bb5b5f0134f028bde638d54ab95443fd2b65ad755276 not found: ID does not exist"
Jul 28 17:58:17.964274 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.964251    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:58:17.969095 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:17.969075    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-80e89-predictor-f86d55dd6-56mdh"]
Jul 28 17:58:18.357707 ip-10-0-134-28 kubenswrapper[2576]: I0728 17:58:18.357678    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" path="/var/lib/kubelet/pods/23af64eb-22ef-421f-8f81-536bada20b4e/volumes"
Jul 28 18:05:51.420615 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.420581    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 18:05:51.421049 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.420788    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container" containerID="cri-o://2c879bbd640dded0a84961eb739ddf2cb7de2324b79e5716e3102d9698552d2c" gracePeriod=30
Jul 28 18:05:51.421049 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.420843    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kube-rbac-proxy" containerID="cri-o://0bc191f372bf77a3de5d3216db854c09010f8a549c707d800b096433c4d37b9e" gracePeriod=30
Jul 28 18:05:51.490132 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.490095    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 18:05:51.490346 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.490321    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container" containerID="cri-o://87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2" gracePeriod=30
Jul 28 18:05:51.490441 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.490345    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kube-rbac-proxy" containerID="cri-o://d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc" gracePeriod=30
Jul 28 18:05:51.587820 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.587789    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerID="0bc191f372bf77a3de5d3216db854c09010f8a549c707d800b096433c4d37b9e" exitCode=2
Jul 28 18:05:51.587978 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:51.587832    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerDied","Data":"0bc191f372bf77a3de5d3216db854c09010f8a549c707d800b096433c4d37b9e"}
Jul 28 18:05:52.592799 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:52.592762    2576 generic.go:356] "Generic (PLEG): container finished" podID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerID="d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc" exitCode=2
Jul 28 18:05:52.593184 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:52.592833    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerDied","Data":"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"}
Jul 28 18:05:53.653715 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:53.653672    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.38:8643/healthz\": dial tcp 10.132.0.38:8643: connect: connection refused"
Jul 28 18:05:54.603051 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.602991    2576 generic.go:356] "Generic (PLEG): container finished" podID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerID="2c879bbd640dded0a84961eb739ddf2cb7de2324b79e5716e3102d9698552d2c" exitCode=0
Jul 28 18:05:54.603105 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.603066    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerDied","Data":"2c879bbd640dded0a84961eb739ddf2cb7de2324b79e5716e3102d9698552d2c"}
Jul 28 18:05:54.679054 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.679031    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 18:05:54.747698 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.747669    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 18:05:54.789055 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.789026    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config\") pod \"d0ad12a3-5566-454f-9129-57fa4b3e5495\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") "
Jul 28 18:05:54.789055 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.789059    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kz9c\" (UniqueName: \"kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c\") pod \"d0ad12a3-5566-454f-9129-57fa4b3e5495\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") "
Jul 28 18:05:54.789237 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.789190    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls\") pod \"d0ad12a3-5566-454f-9129-57fa4b3e5495\" (UID: \"d0ad12a3-5566-454f-9129-57fa4b3e5495\") "
Jul 28 18:05:54.789460 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.789420    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d0ad12a3-5566-454f-9129-57fa4b3e5495" (UID: "d0ad12a3-5566-454f-9129-57fa4b3e5495"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:54.791104 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.791082    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d0ad12a3-5566-454f-9129-57fa4b3e5495" (UID: "d0ad12a3-5566-454f-9129-57fa4b3e5495"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:54.791176 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.791140    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c" (OuterVolumeSpecName: "kube-api-access-7kz9c") pod "d0ad12a3-5566-454f-9129-57fa4b3e5495" (UID: "d0ad12a3-5566-454f-9129-57fa4b3e5495"). InnerVolumeSpecName "kube-api-access-7kz9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:54.890190 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890138    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config\") pod \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") "
Jul 28 18:05:54.890391 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890209    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") pod \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") "
Jul 28 18:05:54.890391 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890239    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qzrdw\" (UniqueName: \"kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw\") pod \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\" (UID: \"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a\") "
Jul 28 18:05:54.890511 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890430    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:54.890511 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890448    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kz9c\" (UniqueName: \"kubernetes.io/projected/d0ad12a3-5566-454f-9129-57fa4b3e5495-kube-api-access-7kz9c\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:54.890511 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890461    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d0ad12a3-5566-454f-9129-57fa4b3e5495-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:54.890641 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.890612    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" (UID: "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 18:05:54.892362 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.892329    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" (UID: "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 18:05:54.892362 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.892341    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw" (OuterVolumeSpecName: "kube-api-access-qzrdw") pod "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" (UID: "7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a"). InnerVolumeSpecName "kube-api-access-qzrdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 18:05:54.991193 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.991089    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-proxy-tls\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:54.991193 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.991135    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzrdw\" (UniqueName: \"kubernetes.io/projected/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-api-access-qzrdw\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:54.991193 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:54.991146    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-134-28.ec2.internal\" DevicePath \"\""
Jul 28 18:05:55.608753 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.608724    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"
Jul 28 18:05:55.608753 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.608733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt" event={"ID":"d0ad12a3-5566-454f-9129-57fa4b3e5495","Type":"ContainerDied","Data":"a4ccf44a15d0fb516ab37c98e810ab5211a6b8a9afbf6c68ae8093015d110068"}
Jul 28 18:05:55.608983 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.608810    2576 scope.go:117] "RemoveContainer" containerID="0bc191f372bf77a3de5d3216db854c09010f8a549c707d800b096433c4d37b9e"
Jul 28 18:05:55.610266 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.610244    2576 generic.go:356] "Generic (PLEG): container finished" podID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerID="87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2" exitCode=0
Jul 28 18:05:55.610346 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.610289    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"
Jul 28 18:05:55.610346 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.610328    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerDied","Data":"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"}
Jul 28 18:05:55.610438 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.610389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" event={"ID":"7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a","Type":"ContainerDied","Data":"f3d599eeea6a29776c23c72fc997f67ae7ce9ba4c6c1bf61c33d3cffbc931e41"}
Jul 28 18:05:55.618461 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.618424    2576 scope.go:117] "RemoveContainer" containerID="2c879bbd640dded0a84961eb739ddf2cb7de2324b79e5716e3102d9698552d2c"
Jul 28 18:05:55.626281 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.626263    2576 scope.go:117] "RemoveContainer" containerID="d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"
Jul 28 18:05:55.633130 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.633109    2576 scope.go:117] "RemoveContainer" containerID="87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"
Jul 28 18:05:55.635291 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.635271    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 18:05:55.641529 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.641506    2576 scope.go:117] "RemoveContainer" containerID="d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"
Jul 28 18:05:55.641957 ip-10-0-134-28 kubenswrapper[2576]: E0728 18:05:55.641939    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc\": container with ID starting with d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc not found: ID does not exist" containerID="d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"
Jul 28 18:05:55.642015 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.641966    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc"} err="failed to get container status \"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc\": rpc error: code = NotFound desc = could not find container \"d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc\": container with ID starting with d7c9a6c66c1a2986d8923541f9e2486ff79ccae4af2380e1adeaf498bc7f79fc not found: ID does not exist"
Jul 28 18:05:55.642015 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.641986    2576 scope.go:117] "RemoveContainer" containerID="87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"
Jul 28 18:05:55.642090 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.642027    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c0ed1-predictor-76c894557-m64nt"]
Jul 28 18:05:55.642220 ip-10-0-134-28 kubenswrapper[2576]: E0728 18:05:55.642205    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2\": container with ID starting with 87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2 not found: ID does not exist" containerID="87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"
Jul 28 18:05:55.642257 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.642224    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2"} err="failed to get container status \"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2\": rpc error: code = NotFound desc = could not find container \"87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2\": container with ID starting with 87f2dd5b248aaef90ebe440e62c94b32e4cad6f733b69a3102ed7e0b80a8f5d2 not found: ID does not exist"
Jul 28 18:05:55.654593 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.654568    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 18:05:55.658626 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.658596    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.39:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 18:05:55.658736 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:55.658705    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c0ed1-predictor-cdc75bd96-gt2ps"]
Jul 28 18:05:56.358264 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:56.358231    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" path="/var/lib/kubelet/pods/7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a/volumes"
Jul 28 18:05:56.358678 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:05:56.358664    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" path="/var/lib/kubelet/pods/d0ad12a3-5566-454f-9129-57fa4b3e5495/volumes"
Jul 28 18:06:24.084783 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.084698    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-92xdj/must-gather-qhnzb"]
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085053    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085066    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085074    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085079    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085089    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085095    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085107    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085112    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085119    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085124    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085134    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085139    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085143    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085149    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085154    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085159    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085166    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085171    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085176    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085181    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container"
Jul 28 18:06:24.085182 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085192    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085198    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085204    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085209    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085253    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085261    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085266    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13baaa32-4f7e-42e5-8d7f-e5020b670207" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085270    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085275    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="23af64eb-22ef-421f-8f81-536bada20b4e" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085280    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085286    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085292    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ad12a3-5566-454f-9129-57fa4b3e5495" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085297    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df2b1d2-0e0c-47a2-b152-e21b00bc8e1a" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085302    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kserve-container"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085308    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c23fd96-bc7e-4393-8a9d-e5466324a746" containerName="kube-rbac-proxy"
Jul 28 18:06:24.085861 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.085312    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c8be5a8-0cef-4fb8-9289-c32719c96da1" containerName="kube-rbac-proxy"
Jul 28 18:06:24.087949 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.087933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.090949 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.090925    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92xdj\"/\"openshift-service-ca.crt\""
Jul 28 18:06:24.091083 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.091003    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-92xdj\"/\"kube-root-ca.crt\""
Jul 28 18:06:24.092134 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.092113    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-92xdj\"/\"default-dockercfg-5gwzn\""
Jul 28 18:06:24.098475 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.098456    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92xdj/must-gather-qhnzb"]
Jul 28 18:06:24.107268 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.107234    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34e90703-cdf2-4dab-a741-5ac660dd7e23-must-gather-output\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.107364 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.107293    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lp7q\" (UniqueName: \"kubernetes.io/projected/34e90703-cdf2-4dab-a741-5ac660dd7e23-kube-api-access-7lp7q\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.207971 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.207934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34e90703-cdf2-4dab-a741-5ac660dd7e23-must-gather-output\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.208142 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.207988    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lp7q\" (UniqueName: \"kubernetes.io/projected/34e90703-cdf2-4dab-a741-5ac660dd7e23-kube-api-access-7lp7q\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.208345 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.208322    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/34e90703-cdf2-4dab-a741-5ac660dd7e23-must-gather-output\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.217760 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.217735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lp7q\" (UniqueName: \"kubernetes.io/projected/34e90703-cdf2-4dab-a741-5ac660dd7e23-kube-api-access-7lp7q\") pod \"must-gather-qhnzb\" (UID: \"34e90703-cdf2-4dab-a741-5ac660dd7e23\") " pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.396976 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.396893    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92xdj/must-gather-qhnzb"
Jul 28 18:06:24.533780 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.530299    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92xdj/must-gather-qhnzb"]
Jul 28 18:06:24.536589 ip-10-0-134-28 kubenswrapper[2576]: W0728 18:06:24.536558    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34e90703_cdf2_4dab_a741_5ac660dd7e23.slice/crio-d954922360fd149816cb4a6e46c969f25cb25d62aa0ca44ff6ef533ff463b5ab WatchSource:0}: Error finding container d954922360fd149816cb4a6e46c969f25cb25d62aa0ca44ff6ef533ff463b5ab: Status 404 returned error can't find the container with id d954922360fd149816cb4a6e46c969f25cb25d62aa0ca44ff6ef533ff463b5ab
Jul 28 18:06:24.538449 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.538432    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 18:06:24.723562 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:24.723458    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92xdj/must-gather-qhnzb" event={"ID":"34e90703-cdf2-4dab-a741-5ac660dd7e23","Type":"ContainerStarted","Data":"d954922360fd149816cb4a6e46c969f25cb25d62aa0ca44ff6ef533ff463b5ab"}
Jul 28 18:06:25.731092 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:25.731055    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92xdj/must-gather-qhnzb" event={"ID":"34e90703-cdf2-4dab-a741-5ac660dd7e23","Type":"ContainerStarted","Data":"b910aecacc605685cf161da487b7aa0e25b4fbec42bc49bdf9b898bbbf92835c"}
Jul 28 18:06:25.731092 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:25.731100    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92xdj/must-gather-qhnzb" event={"ID":"34e90703-cdf2-4dab-a741-5ac660dd7e23","Type":"ContainerStarted","Data":"3042865f5ae84a7e448dbd35975f4c3e21a3156e974162f695bbed52e49c5753"}
Jul 28 18:06:25.750669 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:25.750615    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-92xdj/must-gather-qhnzb" podStartSLOduration=1.0408341 podStartE2EDuration="1.750598281s" totalImagesPullingTime="709.764181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:06:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:25.749131898 +0000 UTC m=+2985.935725204" watchObservedRunningTime="2026-07-28 18:06:25.750598281 +0000 UTC m=+2985.937191586"
Jul 28 18:06:26.800399 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:26.800370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5swwd_9a6a32c3-2640-4cf8-8a4f-daf938694431/global-pull-secret-syncer/0.log"
Jul 28 18:06:26.990985 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:26.990959    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qg46g_df4df771-c44e-4a2f-a59a-f80f87f2e19c/konnectivity-agent/0.log"
Jul 28 18:06:27.090782 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:27.090696    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-134-28.ec2.internal_f0f6325ce07165ebf2e9979367b54cf4/haproxy/0.log"
Jul 28 18:06:31.205268 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.205237    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-t7j7j_4edee1e3-e11c-4092-8aaf-6e350260ccd4/kube-state-metrics/0.log"
Jul 28 18:06:31.238264 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.238236    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-t7j7j_4edee1e3-e11c-4092-8aaf-6e350260ccd4/kube-rbac-proxy-main/0.log"
Jul 28 18:06:31.270621 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.270515    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-t7j7j_4edee1e3-e11c-4092-8aaf-6e350260ccd4/kube-rbac-proxy-self/0.log"
Jul 28 18:06:31.503404 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.503369    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwsp8_6b1c597a-6421-4dda-ba0b-105daa48681c/node-exporter/0.log"
Jul 28 18:06:31.528728 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.528651    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwsp8_6b1c597a-6421-4dda-ba0b-105daa48681c/kube-rbac-proxy/0.log"
Jul 28 18:06:31.556930 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:31.556906    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fwsp8_6b1c597a-6421-4dda-ba0b-105daa48681c/init-textfile/0.log"
Jul 28 18:06:32.061443 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:32.061413    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xhljh_491c4b66-2a14-4929-86fa-494c5b67e8f3/prometheus-operator/0.log"
Jul 28 18:06:32.094799 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:32.094766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-xhljh_491c4b66-2a14-4929-86fa-494c5b67e8f3/kube-rbac-proxy/0.log"
Jul 28 18:06:32.162347 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:32.162309    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b8fcb596-cdsg8_001f3fa4-e221-4daf-8e6d-c67064648f88/telemeter-client/0.log"
Jul 28 18:06:32.191492 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:32.191443    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b8fcb596-cdsg8_001f3fa4-e221-4daf-8e6d-c67064648f88/reload/0.log"
Jul 28 18:06:32.218841 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:32.218811    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5b8fcb596-cdsg8_001f3fa4-e221-4daf-8e6d-c67064648f88/kube-rbac-proxy/0.log"
Jul 28 18:06:33.798110 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:33.798081    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-448sj_669afc54-8ae9-44f2-bf01-c6d3ba582a62/networking-console-plugin/0.log"
Jul 28 18:06:34.873878 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:34.873842    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"]
Jul 28 18:06:34.878935 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:34.878891    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:34.890835 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:34.890807    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"]
Jul 28 18:06:35.007617 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.007571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-podres\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.007817 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.007661    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d79m6\" (UniqueName: \"kubernetes.io/projected/1f237342-8820-4d7f-9596-9f3eabcf1244-kube-api-access-d79m6\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.007817 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.007739    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-lib-modules\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.007817 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.007772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-sys\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.008005 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.007844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-proc\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108492 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d79m6\" (UniqueName: \"kubernetes.io/projected/1f237342-8820-4d7f-9596-9f3eabcf1244-kube-api-access-d79m6\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108711 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108527    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-lib-modules\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108711 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108585    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-sys\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108711 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-proc\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108711 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-podres\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108921 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-proc\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108921 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108708    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-lib-modules\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108921 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-sys\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.108921 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.108791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1f237342-8820-4d7f-9596-9f3eabcf1244-podres\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.118466 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.118425    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d79m6\" (UniqueName: \"kubernetes.io/projected/1f237342-8820-4d7f-9596-9f3eabcf1244-kube-api-access-d79m6\") pod \"perf-node-gather-daemonset-bnnh5\" (UID: \"1f237342-8820-4d7f-9596-9f3eabcf1244\") " pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.195982 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.195885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.368659 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.368624    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"]
Jul 28 18:06:35.781305 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.781270    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5" event={"ID":"1f237342-8820-4d7f-9596-9f3eabcf1244","Type":"ContainerStarted","Data":"76432219262c1c5798529d3de1154ea74c0ac3274f67d687ab86a9756b0584da"}
Jul 28 18:06:35.781501 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.781329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5" event={"ID":"1f237342-8820-4d7f-9596-9f3eabcf1244","Type":"ContainerStarted","Data":"ffe7b2ff51b98dca01885bafc7836024f7d9de808bd579406245327e8b19052c"}
Jul 28 18:06:35.781501 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.781404    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:35.799734 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:35.799682    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5" podStartSLOduration=1.7996647810000002 podStartE2EDuration="1.799664781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:06:35.799038543 +0000 UTC m=+2995.985631839" watchObservedRunningTime="2026-07-28 18:06:35.799664781 +0000 UTC m=+2995.986258130"
Jul 28 18:06:36.490788 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:36.490760    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnqnd_988ee349-bc90-4a64-9787-a4b7688e5f19/dns/0.log"
Jul 28 18:06:36.515645 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:36.515608    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jnqnd_988ee349-bc90-4a64-9787-a4b7688e5f19/kube-rbac-proxy/0.log"
Jul 28 18:06:36.626872 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:36.626847    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-647fs_8267b0b1-b622-489a-878d-6061b47da927/dns-node-resolver/0.log"
Jul 28 18:06:37.241791 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:37.241756    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-w5blq_b05ffd10-3b28-4ae6-9fd5-89b15f0f4139/node-ca/0.log"
Jul 28 18:06:38.099893 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:38.099866    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-84996d74f-sgkml_0d3cc9dd-9a09-4852-8f44-50c87d93c575/router/0.log"
Jul 28 18:06:38.518622 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:38.518598    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mrwds_e434feb5-2d72-4980-a231-12e3ca1449c2/serve-healthcheck-canary/0.log"
Jul 28 18:06:39.145006 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:39.144975    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7nmc_465c3991-57e5-4a79-8a94-78763ce4180e/kube-rbac-proxy/0.log"
Jul 28 18:06:39.171750 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:39.171719    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7nmc_465c3991-57e5-4a79-8a94-78763ce4180e/exporter/0.log"
Jul 28 18:06:39.199886 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:39.199862    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f7nmc_465c3991-57e5-4a79-8a94-78763ce4180e/extractor/0.log"
Jul 28 18:06:41.796194 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:41.796163    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-92xdj/perf-node-gather-daemonset-bnnh5"
Jul 28 18:06:41.799104 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:41.799076    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-7k9s2_90ba8e68-8771-4d00-bba5-96d640379eb7/manager/0.log"
Jul 28 18:06:41.898217 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:41.898188    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-fv9lk_03dae415-ddff-4cc6-8000-7db321a248aa/seaweedfs/0.log"
Jul 28 18:06:48.354342 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.354310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/kube-multus-additional-cni-plugins/0.log"
Jul 28 18:06:48.385537 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.385508    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/egress-router-binary-copy/0.log"
Jul 28 18:06:48.416728 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.416699    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/cni-plugins/0.log"
Jul 28 18:06:48.448687 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.448659    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/bond-cni-plugin/0.log"
Jul 28 18:06:48.478071 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.478034    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/routeoverride-cni/0.log"
Jul 28 18:06:48.512854 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.512831    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/whereabouts-cni-bincopy/0.log"
Jul 28 18:06:48.546648 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.546619    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-g2rl4_07acdd3f-1248-4bb5-ab18-ad1cdf17f694/whereabouts-cni/0.log"
Jul 28 18:06:48.887428 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:48.887395    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-q24qv_02612ec5-b216-495d-92ab-3569ec0ee16e/kube-multus/0.log"
Jul 28 18:06:49.016783 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:49.016733    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tqq8n_4b5333fe-432d-40d7-a893-35a95b16181f/network-metrics-daemon/0.log"
Jul 28 18:06:49.043253 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:49.043229    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tqq8n_4b5333fe-432d-40d7-a893-35a95b16181f/kube-rbac-proxy/0.log"
Jul 28 18:06:50.454904 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.454878    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/ovn-controller/0.log"
Jul 28 18:06:50.477536 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.477514    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/ovn-acl-logging/0.log"
Jul 28 18:06:50.506050 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.506025    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/kube-rbac-proxy-node/0.log"
Jul 28 18:06:50.529653 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.529632    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 18:06:50.552164 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.552141    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/northd/0.log"
Jul 28 18:06:50.576272 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.576251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/nbdb/0.log"
Jul 28 18:06:50.601650 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.601607    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/sbdb/0.log"
Jul 28 18:06:50.654466 ip-10-0-134-28 kubenswrapper[2576]: I0728 18:06:50.654440    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-vcdd5_d37aad0c-f620-44a7-9396-a8cdf7379c15/ovnkube-controller/0.log"