Jul 16 17:32:18.053394 ip-10-0-143-77 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 17:32:18.053404 ip-10-0-143-77 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 17:32:18.053411 ip-10-0-143-77 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 17:32:18.053707 ip-10-0-143-77 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 17:32:28.214882 ip-10-0-143-77 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 17:32:28.214900 ip-10-0-143-77 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot e6713e4d0692445598b00cb8cca5435f --
Jul 16 17:34:29.462732 ip-10-0-143-77 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 17:34:29.833726 ip-10-0-143-77 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:29.833726 ip-10-0-143-77 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 17:34:29.833726 ip-10-0-143-77 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:29.833726 ip-10-0-143-77 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 17:34:29.833726 ip-10-0-143-77 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 17:34:29.836129 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.836043    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 17:34:29.840656 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840639    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.840656 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840656    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.840656 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840659    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840663    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840666    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840669    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840671    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840674    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840678    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840681    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840683    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840691    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840694    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840696    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840699    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840702    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840704    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840707    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840709    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840712    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840714    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840717    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.840759 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840719    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840722    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840725    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840727    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840730    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840732    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840735    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840738    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840740    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840742    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840745    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840748    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840751    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840754    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840757    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840759    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840762    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840767    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840771    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.841237 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840774    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840777    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840779    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840782    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840785    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840787    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840790    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840792    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840795    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840797    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840799    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840802    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840804    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840806    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840809    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840812    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840815    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840817    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840820    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840822    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.841736 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840824    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840827    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840830    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840832    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840835    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840837    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840840    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840843    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840845    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840848    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840851    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840853    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840857    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840860    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840863    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840865    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840868    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840871    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840873    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840876    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.842214 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840878    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840881    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840890    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840894    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840897    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840901    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840904    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840906    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840910    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840914    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840916    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840919    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840921    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840924    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840927    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840930    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840932    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.840935    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841657    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.842716 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841664    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841667    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841670    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841672    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841681    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841684    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841687    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841689    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841692    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841694    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841697    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841700    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841702    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841704    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841707    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841709    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841712    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841715    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841717    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841720    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.843163 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841722    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841725    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841727    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841730    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841732    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841734    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841737    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841739    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841742    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841745    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841747    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841751    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841753    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841757    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841761    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841764    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841766    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841772    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841775    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841777    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.843664 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841779    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841782    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841784    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841786    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841789    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841791    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841793    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841796    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841798    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841801    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841803    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841806    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841808    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841810    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841813    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841817    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841819    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841822    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841825    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841827    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.844142 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841830    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841833    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841835    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841838    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841841    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841843    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841846    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841849    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841851    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841854    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841857    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841859    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841861    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841864    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841866    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841868    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841871    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841873    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841875    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841878    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.844631 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841880    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841884    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841887    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841890    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841892    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841896    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841898    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841901    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841903    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841906    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841908    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841911    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841913    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841916    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841918    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841921    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841924    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.841926    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842319    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.845138 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842323    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842326    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842329    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842332    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842334    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842336    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842339    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842341    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842344    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842346    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842348    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842351    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842353    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842355    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842360    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842363    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842366    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842369    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842372    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.845612 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842375    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842377    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842381    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842383    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842386    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842389    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842391    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842394    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842396    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842399    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842402    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842404    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842407    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842409    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842411    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842414    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842417    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842419    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842422    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842424    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.846062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842427    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842429    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842446    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842449    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842451    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842454    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842456    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842459    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842461    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842464    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842467    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842470    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842472    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842475    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842477    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842480    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842483    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842486    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842488    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842492    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.846585 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842494    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842498    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842500    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842502    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842505    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842507    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842509    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842512    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842514    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842516    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842519    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842522    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842524    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842527    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842531    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842534    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842537    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842540    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842542    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.847062 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842545    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842548    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842551    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842554    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842557    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842560    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842562    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842565    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842568    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842573    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842575    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842578    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842580    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842583    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842585    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842588    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842590    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842593    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842595    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.842598    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.847534 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842677    2576 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842687    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842699    2576 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842705    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842713    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842718    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842724    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842728    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842732    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842735    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842738    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842741    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842744    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842747    2576 flags.go:64] FLAG: --cgroup-root=""
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842750    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842753    2576 flags.go:64] FLAG: --client-ca-file=""
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842756    2576 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842759    2576 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842763    2576 flags.go:64] FLAG: --cluster-domain=""
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842766    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842769    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842774    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842778    2576 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842781    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 17:34:29.848048 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842785    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842788    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842791    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842794    2576 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842797    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842800    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842803    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842805    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842810    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842813    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842816    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842819    2576 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842822    2576 flags.go:64] FLAG: --enable-server="true"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842825    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842830    2576 flags.go:64] FLAG: --event-burst="100"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842833    2576 flags.go:64] FLAG: --event-qps="50"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842835    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842838    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842841    2576 flags.go:64] FLAG: --eviction-hard=""
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842845    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842848    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842851    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842854    2576 flags.go:64] FLAG: --eviction-soft=""
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842856    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842859    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 17:34:29.848669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842862    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842865    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842868    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842870    2576 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842875    2576 flags.go:64] FLAG: --feature-gates=""
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842879    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842882    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842884    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842887    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842890    2576 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842893    2576 flags.go:64] FLAG: --help="false"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842896    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-143-77.ec2.internal"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842899    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842902    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842904    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842908    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842911    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842913    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842916    2576 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842919    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842922    2576 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842926    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842929    2576 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842932    2576 flags.go:64] FLAG: --kube-reserved=""
Jul 16 17:34:29.849305 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842935    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842937    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842940    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842943    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842946    2576 flags.go:64] FLAG: --lock-file=""
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842948    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842951    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842954    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842959    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842962    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842967    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842970    2576 flags.go:64] FLAG: --logging-format="text"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842975    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842978    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842981    2576 flags.go:64] FLAG: --manifest-url=""
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842984    2576 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842988    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842991    2576 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842995    2576 flags.go:64] FLAG: --max-pods="110"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.842998    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843001    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843004    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843007    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843009    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843012    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 17:34:29.849912 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843015    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843023    2576 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843026    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843029    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843032    2576 flags.go:64] FLAG: --pod-cidr=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843035    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843041    2576 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843044    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843047    2576 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843050    2576 flags.go:64] FLAG: --port="10250"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843053    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843055    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06de048d65c2ddf12"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843059    2576 flags.go:64] FLAG: --qos-reserved=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843061    2576 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843064    2576 flags.go:64] FLAG: --register-node="true"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843067    2576 flags.go:64] FLAG: --register-with-taints=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843070    2576 flags.go:64] FLAG: --registry-burst="10"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843074    2576 flags.go:64] FLAG: --registry-qps="5"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843077    2576 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843079    2576 flags.go:64] FLAG: --reserved-memory=""
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843085    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843088    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843091    2576 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843093    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843096    2576 flags.go:64] FLAG: --runonce="false"
Jul 16 17:34:29.850523 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843099    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843102    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843105    2576 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843107    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843110    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843113    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843116    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843119    2576 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843121    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843124    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843127    2576 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843130    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843132    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843136    2576 flags.go:64] FLAG: --system-cgroups=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843138    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843143    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843146    2576 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843149    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843152    2576 flags.go:64] FLAG: --tls-min-version=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843155    2576 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843157    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843160    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843163    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843166    2576 flags.go:64] FLAG: --v="2"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843171    2576 flags.go:64] FLAG: --version="false"
Jul 16 17:34:29.851127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843175    2576 flags.go:64] FLAG: --vmodule=""
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843179    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843183    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843331    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843335    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843339    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843342    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843345    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843347    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843350    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843352    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843355    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843359    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843362    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843365    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843368    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843371    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843374    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843376    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.851843 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843379    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843381    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843386    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843388    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843391    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843394    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843396    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843399    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843401    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843404    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843406    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843408    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843412    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843414    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843417    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843420    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843423    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843425    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843428    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843446    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.852299 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843450    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843452    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843455    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843457    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843459    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843462    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843465    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843467    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843470    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843472    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843475    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843477    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843479    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843482    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843484    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843487    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843490    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843493    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843495    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843498    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.853087 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843500    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843502    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843505    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843507    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843511    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843514    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843516    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843520    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843522    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843526    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843529    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843532    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843534    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843536    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843539    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843541    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843544    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843546    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843549    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.853710 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843551    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843553    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843556    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843558    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843560    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843563    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843565    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843568    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843571    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843574    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843576    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843579    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843581    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843583    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843586    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843588    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843590    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843594    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843596    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843599    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.854161 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843602    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843604    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843607    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.843609    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.843615    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.850350    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.850365    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.850384    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.850390    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850456    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850462    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850465    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850468    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850470    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850473    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.854677 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850475    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850478    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850480    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850483    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850485    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850488    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850490    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850493    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850496    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850498    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850501    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850503    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850506    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850508    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850511    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850513    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850515    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850519    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850523    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.855151 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850526    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850529    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850532    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850535    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850545    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850549    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850552    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850554    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850557    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850560    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850563    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850565    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850568    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850570    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850572    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850575    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850577    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850580    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850582    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850585    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.855625 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850587    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850590    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850592    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850595    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850599    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850602    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850604    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850607    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850610    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850612    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850614    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850617    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850620    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850623    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850627    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850629    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850633    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850636    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850639    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850641    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.856107 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850644    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850646    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850649    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850652    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850654    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850656    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850659    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850661    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850664    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850667    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850669    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850671    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850674    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850676    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850679    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850681    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850684    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850686    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850688    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.856680 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850691    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850693    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850695    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850698    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850700    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850703    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850705    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850707    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850710    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850713    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850716    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850718    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850721    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850723    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850726    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.857159 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.850730    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850837    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850842    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850844    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850847    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850850    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850853    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850856    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850859    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850861    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850864    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850866    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850868    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850871    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850874    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850876    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850879    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850881    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850884    2576 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850886    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850888    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 17:34:29.857542 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850891    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850893    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850895    2576 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850897    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850900    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850903    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850905    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850908    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850911    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850915    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850919    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850922    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850924    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850927    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850930    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850932    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850935    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850937    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850939    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850942    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 17:34:29.858045 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850944    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850947    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850949    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850951    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850953    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850956    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850959    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850961    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850963    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850965    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850968    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850970    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850972    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850975    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850977    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850981    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850984    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850987    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850990    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 17:34:29.858563 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850993    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850996    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.850999    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851002    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851004    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851007    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851009    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851012    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851014    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851017    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851019    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851021    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851024    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851026    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851028    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851031    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851033    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851036    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851038    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851040    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 17:34:29.859039 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851043    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851045    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851047    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851050    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851052    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851054    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851057    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851059    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851062    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851064    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851067    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851069    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851071    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851074    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851076    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851079    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851081    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851084    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851088    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 17:34:29.859649 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:29.851091    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.851096    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.851199    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.854688    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.856218    2576 server.go:1019] "Starting client certificate rotation"
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.856319    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.856351    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 17:34:29.860177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.858854    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:29.863017 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.863003    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 17:34:29.872607 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.872591    2576 log.go:25] "Validated CRI v1 runtime API"
Jul 16 17:34:29.874520 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.874507    2576 log.go:25] "Validated CRI v1 image API"
Jul 16 17:34:29.876322 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.876307    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 17:34:29.878465 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.878417    2576 fs.go:135] Filesystem UUIDs: map[3322ec65-05cf-46b9-8c2e-7e55b1356334:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f0e69f0d-1532-47c7-bede-79892216021d:/dev/nvme0n1p3]
Jul 16 17:34:29.878545 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.878464    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 17:34:29.883717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.883590    2576 manager.go:216] Machine: {Timestamp:2026-07-16 17:34:29.882290722 +0000 UTC m=+0.306020773 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096629 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29b914d99c48b8e0c46333fcf6bd44 SystemUUID:ec29b914-d99c-48b8-e0c4-6333fcf6bd44 BootID:e6713e4d-0692-4455-98b0-0cb8cca5435f Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:dc:81:91:ad:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:dc:81:91:ad:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:63:92:1c:68:ac Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 17:34:29.883717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.883708    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 17:34:29.883856 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.883790    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 17:34:29.886781 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.886754    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:29.887226 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.887196    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 17:34:29.887424 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.887232    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-77.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 17:34:29.887914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.887901    2576 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 17:34:29.887975 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.887920    2576 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 17:34:29.887975 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.887938    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:29.888729 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.888716    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 17:34:29.889807 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.889795    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 17:34:29.892465 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892452    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:29.892611 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892599    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 17:34:29.892665 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892648    2576 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 17:34:29.892665 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892662    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 17:34:29.892790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892681    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 17:34:29.892790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892697    2576 kubelet.go:398] "Adding apiserver pod source"
Jul 16 17:34:29.892790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.892709    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 17:34:29.893394 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.893380    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:29.893509 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.893403    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 17:34:29.896758 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.896716    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 17:34:29.898481 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898464    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 17:34:29.898566 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898504    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 17:34:29.898763 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898749    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 17:34:29.898861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898769    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 17:34:29.898861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898778    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 17:34:29.898861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898788    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 17:34:29.898861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898800    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 17:34:29.898861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.898809    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 17:34:29.899121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899111    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 17:34:29.899167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899124    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 17:34:29.899167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899135    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 17:34:29.899167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899144    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 17:34:29.899167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899156    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 17:34:29.899167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899168    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 17:34:29.899368 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899199    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 17:34:29.899368 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.899208    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 17:34:29.902475 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.902460    2576 server.go:1268] "Started kubelet"
Jul 16 17:34:29.902596 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.902541    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 17:34:29.902705 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.902618    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 17:34:29.902705 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.902551    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 17:34:29.903269 ip-10-0-143-77 systemd[1]: Started Kubernetes Kubelet.
Jul 16 17:34:29.903901 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.903877    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-77.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 17:34:29.904017 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.903916    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 17:34:29.904498 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.904473    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 17:34:29.904575 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.904473    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 17:34:29.905259 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.905245    2576 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 17:34:29.907477 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.907459    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 17:34:29.907687 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.906800    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5a8502cf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.902418639 +0000 UTC m=+0.326148690,LastTimestamp:2026-07-16 17:34:29.902418639 +0000 UTC m=+0.326148690,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.909374 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.909362    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 17:34:29.909466 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.909446    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:29.909601 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.909576    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:29.909938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.909917    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 17:34:29.910014 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.909918    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 17:34:29.910014 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.909961    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 17:34:29.910088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910047    2576 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 17:34:29.910088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910055    2576 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 17:34:29.910530 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910498    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 17:34:29.910530 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910531    2576 factory.go:55] Registering systemd factory
Jul 16 17:34:29.910648 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910538    2576 factory.go:223] Registration of the systemd container factory successfully
Jul 16 17:34:29.910780 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910766    2576 factory.go:153] Registering CRI-O factory
Jul 16 17:34:29.910868 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910783    2576 factory.go:223] Registration of the crio container factory successfully
Jul 16 17:34:29.910868 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910807    2576 factory.go:103] Registering Raw factory
Jul 16 17:34:29.910868 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.910823    2576 manager.go:1196] Started watching for new ooms in manager
Jul 16 17:34:29.911034 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.911010    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-77.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 17:34:29.911200 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.911181    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 17:34:29.912538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.911856    2576 manager.go:349] Starting recovery of all containers
Jul 16 17:34:29.922965 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.922823    2576 manager.go:354] Recovery completed
Jul 16 17:34:29.928774 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.928760    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:29.931317 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931300    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:29.931384 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931329    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:29.931384 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931340    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:29.931927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931909    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 17:34:29.932004 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931932    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 17:34:29.932004 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931943    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:29.932004 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.931954    2576 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 17:34:29.932683 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.932667    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:29.932765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.932694    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:29.932765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.932704    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:29.933272 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.933259    2576 policy_none.go:49] "None policy: Start"
Jul 16 17:34:29.933320 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.933276    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 17:34:29.933320 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.933288    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 17:34:29.934521 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.934509    2576 policy_none.go:47] "Start"
Jul 16 17:34:29.944609 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.944545    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3df5d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931316692 +0000 UTC m=+0.355046742,LastTimestamp:2026-07-16 17:34:29.931316692 +0000 UTC m=+0.355046742,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.952616 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.952523    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3e3a3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931334205 +0000 UTC m=+0.355064256,LastTimestamp:2026-07-16 17:34:29.931334205 +0000 UTC m=+0.355064256,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.967297 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.967225    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3e67cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931345867 +0000 UTC m=+0.355075919,LastTimestamp:2026-07-16 17:34:29.931345867 +0000 UTC m=+0.355075919,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.971572 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.971558    2576 manager.go:337] "Starting Device Plugin manager"
Jul 16 17:34:29.971648 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.971587    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 17:34:29.971648 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.971596    2576 server.go:87] "Starting device plugin registration server"
Jul 16 17:34:29.971885 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.971871    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 17:34:29.971978 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.971886    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 17:34:29.971978 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.971973    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 17:34:29.972100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.972088    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 17:34:29.972152 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.972109    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 17:34:29.972590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.972572    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 17:34:29.972660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.972618    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:29.976578 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.976517    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-77.ec2.internal.18c2d59c5c3df5d4\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3df5d4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931316692 +0000 UTC m=+0.355046742,LastTimestamp:2026-07-16 17:34:29.93268184 +0000 UTC m=+0.356411891,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.983622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.983597    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kkrwd"
Jul 16 17:34:29.985515 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.985454    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-77.ec2.internal.18c2d59c5c3e3a3d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3e3a3d  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931334205 +0000 UTC m=+0.355064256,LastTimestamp:2026-07-16 17:34:29.932698916 +0000 UTC m=+0.356428968,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:29.992971 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:29.992955    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kkrwd"
Jul 16 17:34:29.994322 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:29.994249    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-143-77.ec2.internal.18c2d59c5c3e67cb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-77.ec2.internal.18c2d59c5c3e67cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-77.ec2.internal,UID:ip-10-0-143-77.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-143-77.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-143-77.ec2.internal,},FirstTimestamp:2026-07-16 17:34:29.931345867 +0000 UTC m=+0.355075919,LastTimestamp:2026-07-16 17:34:29.932707952 +0000 UTC m=+0.356438003,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-77.ec2.internal,}"
Jul 16 17:34:30.072191 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.072164    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:30.073395 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.073034    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:30.073395 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.073062    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:30.073395 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.073071    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:30.073395 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.073102    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.081622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.081599    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.081622 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.081622    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-77.ec2.internal\": node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.083752 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.083697    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 17:34:30.085225 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.085202    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 17:34:30.085302 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.085248    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 17:34:30.085302 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.085273    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 17:34:30.085399 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.085314    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 17:34:30.087177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.087112    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:30.096612 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.096597    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.185689 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.185658    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"]
Jul 16 17:34:30.185822 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.185740    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:30.186740 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.186721    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:30.186740 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.186752    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:30.186887 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.186761    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:30.188673 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.188660    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:30.188824 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.188811    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.188868 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.188840    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:30.189381 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189364    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:30.189381 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189380    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:30.189506 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189398    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:30.189506 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189399    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:30.189506 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189421    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:30.189506 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.189408    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:30.191062 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.191046    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.191128 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.191072    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 17:34:30.191794 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.191775    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 17:34:30.191867 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.191805    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 17:34:30.191867 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.191814    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeHasSufficientPID"
Jul 16 17:34:30.197472 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.197457    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.208018 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.207998    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-77.ec2.internal\" not found" node="ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.212032 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.212015    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d6c1142d0e1d3e99bb10c691cfa84-config\") pod \"kube-apiserver-proxy-ip-10-0-143-77.ec2.internal\" (UID: \"648d6c1142d0e1d3e99bb10c691cfa84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.212111 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.212039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.212111 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.212062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.212111 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.212064    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-77.ec2.internal\" not found" node="ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.297815 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.297788    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.312502 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d6c1142d0e1d3e99bb10c691cfa84-config\") pod \"kube-apiserver-proxy-ip-10-0-143-77.ec2.internal\" (UID: \"648d6c1142d0e1d3e99bb10c691cfa84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.312568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312509    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.312568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312525    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.312632 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312570    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.312632 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312575    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/648d6c1142d0e1d3e99bb10c691cfa84-config\") pod \"kube-apiserver-proxy-ip-10-0-143-77.ec2.internal\" (UID: \"648d6c1142d0e1d3e99bb10c691cfa84\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.312632 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.312611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/910cf08b8d7e1547a1739427e869cca2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal\" (UID: \"910cf08b8d7e1547a1739427e869cca2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.398785 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.398716    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.499285 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.499260    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.510468 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.510453    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.515223 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.515208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:30.600041 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.599996    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.702515 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.702483    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.804218 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.804185    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.857753 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.857721    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 17:34:30.858182 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.857872    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 17:34:30.904380 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:30.904344    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-77.ec2.internal\" not found"
Jul 16 17:34:30.978177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.978097    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:30.995614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.995581    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 17:29:30 +0000 UTC" deadline="2028-04-08 18:16:47.466883686 +0000 UTC"
Jul 16 17:34:30.995614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:30.995612    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15168h42m16.471274804s"
Jul 16 17:34:31.009761 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.009733    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:31.020229 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.020210    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:31.022025 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.022003    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal"
Jul 16 17:34:31.033621 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.033602    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 17:34:31.154323 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.154300    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:31.155260 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:31.155227    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648d6c1142d0e1d3e99bb10c691cfa84.slice/crio-9de425906691b98bab5ec27258dfff9df3caa52177350d696dba8d2a1f4eb8f9 WatchSource:0}: Error finding container 9de425906691b98bab5ec27258dfff9df3caa52177350d696dba8d2a1f4eb8f9: Status 404 returned error can't find the container with id 9de425906691b98bab5ec27258dfff9df3caa52177350d696dba8d2a1f4eb8f9
Jul 16 17:34:31.159374 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.159358    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:34:31.387768 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.387686    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:31.857191 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.857104    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 17:34:31.895317 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.895289    2576 apiserver.go:52] "Watching apiserver"
Jul 16 17:34:31.902969 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.902949    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 17:34:31.903936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.903911    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal","openshift-multus/multus-vcz5q","openshift-multus/network-metrics-daemon-48hrx","openshift-network-diagnostics/network-check-target-fqbvg","openshift-network-operator/iptables-alerter-8s7qq","kube-system/konnectivity-agent-n2sbn","kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr","openshift-cluster-node-tuning-operator/tuned-ljskh","openshift-image-registry/node-ca-nx72m","openshift-multus/multus-additional-cni-plugins-wlj77","openshift-ovn-kubernetes/ovnkube-node-ktrzs","kube-system/global-pull-secret-syncer-4z97z","openshift-dns/node-resolver-qzlzg"]
Jul 16 17:34:31.906020 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.906005    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:31.907194 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.907177    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:31.907271 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:31.907242    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:31.908300 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.908281    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:31.908374 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:31.908343    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:31.908614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.908599    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.908674 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.908621    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rnn8c\""
Jul 16 17:34:31.908876 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.908863    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.909219 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.909207    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 17:34:31.909468 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.909433    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:31.909570 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:31.909507    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:31.910075 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.910062    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 17:34:31.910761 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.910729    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:31.912134 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.912113    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:31.913036 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.913018    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.913111 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.913024    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 17:34:31.913376 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.913359    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:31.913474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.913388    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tc9lt\""
Jul 16 17:34:31.913474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.913422    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.914813 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.914791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7sbjq\""
Jul 16 17:34:31.914900 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.914832    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:31.914962 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.914925    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 17:34:31.916052 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.916033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:31.917816 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.917797    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:31.918674 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918653    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 17:34:31.918674 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918673    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.918871 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918673    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.918871 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918797    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.918871 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gnqt5\""
Jul 16 17:34:31.919085 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.918943    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 17:34:31.919313 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.919419 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919397    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.919512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919417    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fwp9x\""
Jul 16 17:34:31.919573 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919523    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-mj5qw\""
Jul 16 17:34:31.919764 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919747    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.919835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:31.919895 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.919858    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 17:34:31.920336 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920315    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.920540 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920519    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wlsqw\""
Jul 16 17:34:31.920634 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920617    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 17:34:31.920700 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-dbus\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:31.920700 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920683    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-iptables-alerter-script\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:31.920801 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-host-slash\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:31.920801 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920757    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qjn6\" (UniqueName: \"kubernetes.io/projected/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-kube-api-access-7qjn6\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:31.920801 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:31.920801 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmwd9\" (UniqueName: \"kubernetes.io/projected/f861f763-4c0a-4bfa-9880-aabe7948600b-kube-api-access-pmwd9\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:31.920982 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920828    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.920982 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920841    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-kubelet-config\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:31.920982 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:31.920982 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920900    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2625d2-c2e2-4d64-a7b2-e5def0303940-host\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:31.920982 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.920975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtznj\" (UniqueName: \"kubernetes.io/projected/5e2625d2-c2e2-4d64-a7b2-e5def0303940-kube-api-access-qtznj\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:31.921162 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.921006    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:31.921162 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.921038    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2625d2-c2e2-4d64-a7b2-e5def0303940-serviceca\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:31.921781 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.921766    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:31.923268 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.923248    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 17:34:31.923352 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.923339    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 17:34:31.923399 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.923367    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-44lmx\""
Jul 16 17:34:31.924360 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.924343    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rh97j\""
Jul 16 17:34:31.924427 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.924372    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 17:34:31.927887 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.927864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 17:34:31.928008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.927890    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 17:34:31.928008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.927954    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 17:34:31.928160 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.928143    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 17:34:31.928366 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.928330    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 17:34:31.928366 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.928334    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 17:34:31.954361 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.954343    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tf8cz"
Jul 16 17:34:31.964728 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:31.964710    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tf8cz"
Jul 16 17:34:32.011578 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.011555    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 17:34:32.021410 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-var-lib-kubelet\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.021584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99rld\" (UniqueName: \"kubernetes.io/projected/83464c6d-46a8-4fd7-a964-25bdc629e973-kube-api-access-99rld\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.021584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-system-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.021584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021493    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.021584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysconfig\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.021791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021630    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-run\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.021791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-sys\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.021791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-socket-dir-parent\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.021791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2625d2-c2e2-4d64-a7b2-e5def0303940-host\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.021791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021727    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-tmp\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-multus\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021843    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckhl\" (UniqueName: \"kubernetes.io/projected/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-kube-api-access-4ckhl\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e2625d2-c2e2-4d64-a7b2-e5def0303940-host\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f116-f57c-42b7-aa3e-06d562f2cc05-konnectivity-ca\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021938    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-slash\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021971    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.021993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-device-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.022040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d83f116-f57c-42b7-aa3e-06d562f2cc05-agent-certs\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022056    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-cnibin\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-netns\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022104    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-etc-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022125    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-ovn\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-host\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qjn6\" (UniqueName: \"kubernetes.io/projected/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-kube-api-access-7qjn6\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-os-release\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022283    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-node-log\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022309    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx2hf\" (UniqueName: \"kubernetes.io/projected/393a0ebb-5874-4cec-b039-b84de1526fb4-kube-api-access-jx2hf\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-tuned\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022340    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022355    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cnibin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022372    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-system-cni-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022510    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtznj\" (UniqueName: \"kubernetes.io/projected/5e2625d2-c2e2-4d64-a7b2-e5def0303940-kube-api-access-qtznj\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.022568    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.022647    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:32.522628479 +0000 UTC m=+2.946358517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e969db-03bb-4ef2-8053-4e85ee82b946-tmp-dir\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022789    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-bin\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022821    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022848    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-conf-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022872    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.022913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022897    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-systemd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-dbus\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-multus-certs\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.022995    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-etc-kubernetes\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-var-lib-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-script-lib\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-kubelet-config\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023148    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-dbus\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-systemd\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023196    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/854d98e3-464a-4550-b809-e0964d0c8175-kubelet-config\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023240    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-config\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023270    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023297    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-systemd-units\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-env-overrides\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2625d2-c2e2-4d64-a7b2-e5def0303940-serviceca\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cni-binary-copy\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.023538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023398    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-daemon-config\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3e969db-03bb-4ef2-8053-4e85ee82b946-hosts-file\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023445    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjhx\" (UniqueName: \"kubernetes.io/projected/f3e969db-03bb-4ef2-8053-4e85ee82b946-kube-api-access-tqjhx\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023488    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023505    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023555    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-conf\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-bin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-hostroot\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023657    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czz9c\" (UniqueName: \"kubernetes.io/projected/30488cdf-7b40-4e46-ab07-b94cb7706253-kube-api-access-czz9c\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-netd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/393a0ebb-5874-4cec-b039-b84de1526fb4-ovn-node-metrics-cert\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023747    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmwd9\" (UniqueName: \"kubernetes.io/projected/f861f763-4c0a-4bfa-9880-aabe7948600b-kube-api-access-pmwd9\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023807    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-netns\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9xsf\" (UniqueName: \"kubernetes.io/projected/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kube-api-access-j9xsf\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.024416 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023936    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-modprobe-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.023989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-os-release\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-k8s-cni-cncf-io\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024072    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024100    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-kubelet\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-kubelet\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024197    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-log-socket\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024230    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-iptables-alerter-script\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.024238    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024262    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-host-slash\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.024295    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:32.524277096 +0000 UTC m=+2.948007150 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024317    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-kubernetes\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-host-slash\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-lib-modules\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.025143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024564    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/5e2625d2-c2e2-4d64-a7b2-e5def0303940-serviceca\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.025755 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.024954    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-iptables-alerter-script\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.029819 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.029799    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:32.029819 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.029828    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:32.029994 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.029842    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:32.029994 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.029910    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:32.52989347 +0000 UTC m=+2.953623532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:32.030103 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.030074    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 17:34:32.033858 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.033834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtznj\" (UniqueName: \"kubernetes.io/projected/5e2625d2-c2e2-4d64-a7b2-e5def0303940-kube-api-access-qtznj\") pod \"node-ca-nx72m\" (UID: \"5e2625d2-c2e2-4d64-a7b2-e5def0303940\") " pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.033858 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.033850    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qjn6\" (UniqueName: \"kubernetes.io/projected/96ac4f84-129c-48f7-bd5f-0f661b0f80b8-kube-api-access-7qjn6\") pod \"iptables-alerter-8s7qq\" (UID: \"96ac4f84-129c-48f7-bd5f-0f661b0f80b8\") " pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.034482 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.034439    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmwd9\" (UniqueName: \"kubernetes.io/projected/f861f763-4c0a-4bfa-9880-aabe7948600b-kube-api-access-pmwd9\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:32.090298 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.090242    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal" event={"ID":"648d6c1142d0e1d3e99bb10c691cfa84","Type":"ContainerStarted","Data":"9de425906691b98bab5ec27258dfff9df3caa52177350d696dba8d2a1f4eb8f9"}
Jul 16 17:34:32.124835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-multus-certs\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.124835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-etc-kubernetes\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.124835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124848    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-var-lib-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-script-lib\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-multus-certs\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124906    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-etc-kubernetes\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-systemd\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-systemd\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124951    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-var-lib-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-config\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.124992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-systemd-units\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125064    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-env-overrides\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125098    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cni-binary-copy\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-daemon-config\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125321    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-systemd-units\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3e969db-03bb-4ef2-8053-4e85ee82b946-hosts-file\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125404    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjhx\" (UniqueName: \"kubernetes.io/projected/f3e969db-03bb-4ef2-8053-4e85ee82b946-kube-api-access-tqjhx\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125432    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125587    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-conf\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-bin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-script-lib\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-hostroot\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-env-overrides\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f3e969db-03bb-4ef2-8053-4e85ee82b946-hosts-file\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-daemon-config\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.125881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.126548 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125893    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.126862 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.126838    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-conf\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.126961 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.126889    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysctl-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.127206 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127171    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.127287 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127214    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/393a0ebb-5874-4cec-b039-b84de1526fb4-ovnkube-config\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.127287 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.125766    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czz9c\" (UniqueName: \"kubernetes.io/projected/30488cdf-7b40-4e46-ab07-b94cb7706253-kube-api-access-czz9c\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.127287 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127259    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-hostroot\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.127438 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127287    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-netd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.127438 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127323    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/393a0ebb-5874-4cec-b039-b84de1526fb4-ovn-node-metrics-cert\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.127438 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127361    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-netns\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.127438 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127391    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9xsf\" (UniqueName: \"kubernetes.io/projected/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kube-api-access-j9xsf\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.127438 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-modprobe-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127485    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-os-release\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127541    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-k8s-cni-cncf-io\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127567    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127632    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-kubelet\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.127714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-kubelet\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-log-socket\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127754    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-kubernetes\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127784    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-lib-modules\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-modprobe-d\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127970    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-lib-modules\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-var-lib-kubelet\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128015    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99rld\" (UniqueName: \"kubernetes.io/projected/83464c6d-46a8-4fd7-a964-25bdc629e973-kube-api-access-99rld\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128049    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-system-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128056    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-os-release\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128088    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-k8s-cni-cncf-io\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128117    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysconfig\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.128164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128169    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128185    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-sysconfig\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-run\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128240    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-sys\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128271    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-socket-dir-parent\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-tmp\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128341    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-multus\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckhl\" (UniqueName: \"kubernetes.io/projected/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-kube-api-access-4ckhl\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f116-f57c-42b7-aa3e-06d562f2cc05-konnectivity-ca\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cni-binary-copy\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-slash\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128503    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128554    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-device-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128618    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d83f116-f57c-42b7-aa3e-06d562f2cc05-agent-certs\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.127323    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-bin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-cnibin\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128694    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-run-netns\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.128810 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128682    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-netns\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-etc-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128884    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-netns\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-ovn\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128968    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-var-lib-kubelet\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128978    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-host\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-host\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129095    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-kubelet\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-log-socket\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129138    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-kubelet\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129165    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-kubernetes\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-run-ovn-kubernetes\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-run\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129222    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129252    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-ovn\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129259    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-os-release\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.129636 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129324    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/83464c6d-46a8-4fd7-a964-25bdc629e973-sys\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-etc-openvswitch\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129373    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-etc-selinux\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129406    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-node-log\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-system-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx2hf\" (UniqueName: \"kubernetes.io/projected/393a0ebb-5874-4cec-b039-b84de1526fb4-kube-api-access-jx2hf\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-os-release\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129498    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-socket-dir-parent\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129492    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-tuned\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-device-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-slash\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129855    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-socket-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.129930    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-host-var-lib-cni-multus\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0d83f116-f57c-42b7-aa3e-06d562f2cc05-konnectivity-ca\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-registration-dir\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130305    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-node-log\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.130533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130552    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-cnibin\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.128240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-netd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.130952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cnibin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-system-cni-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131101    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e969db-03bb-4ef2-8053-4e85ee82b946-tmp-dir\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131129    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-bin\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/30488cdf-7b40-4e46-ab07-b94cb7706253-cni-binary-copy\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-conf-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131249    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/30488cdf-7b40-4e46-ab07-b94cb7706253-system-cni-dir\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.131342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131279    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-systemd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131350    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-cni-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-run-systemd\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131494    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/393a0ebb-5874-4cec-b039-b84de1526fb4-host-cni-bin\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131579    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-sys-fs\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131627    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-multus-conf-dir\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-cnibin\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.131945 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.131757    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3e969db-03bb-4ef2-8053-4e85ee82b946-tmp-dir\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.133221 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.133120    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0d83f116-f57c-42b7-aa3e-06d562f2cc05-agent-certs\") pod \"konnectivity-agent-n2sbn\" (UID: \"0d83f116-f57c-42b7-aa3e-06d562f2cc05\") " pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.133391 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.133371    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/393a0ebb-5874-4cec-b039-b84de1526fb4-ovn-node-metrics-cert\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.135259 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.135239    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-etc-tuned\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.137318 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.137295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjhx\" (UniqueName: \"kubernetes.io/projected/f3e969db-03bb-4ef2-8053-4e85ee82b946-kube-api-access-tqjhx\") pod \"node-resolver-qzlzg\" (UID: \"f3e969db-03bb-4ef2-8053-4e85ee82b946\") " pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.139044 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.138980    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-99rld\" (UniqueName: \"kubernetes.io/projected/83464c6d-46a8-4fd7-a964-25bdc629e973-kube-api-access-99rld\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.139937 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.139712    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/83464c6d-46a8-4fd7-a964-25bdc629e973-tmp\") pod \"tuned-ljskh\" (UID: \"83464c6d-46a8-4fd7-a964-25bdc629e973\") " pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.141286 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.141260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckhl\" (UniqueName: \"kubernetes.io/projected/06fccf7d-2fcb-47cf-b0f4-3d4f169a1254-kube-api-access-4ckhl\") pod \"multus-vcz5q\" (UID: \"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254\") " pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.141686 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.141662    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx2hf\" (UniqueName: \"kubernetes.io/projected/393a0ebb-5874-4cec-b039-b84de1526fb4-kube-api-access-jx2hf\") pod \"ovnkube-node-ktrzs\" (UID: \"393a0ebb-5874-4cec-b039-b84de1526fb4\") " pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.142751 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.142728    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czz9c\" (UniqueName: \"kubernetes.io/projected/30488cdf-7b40-4e46-ab07-b94cb7706253-kube-api-access-czz9c\") pod \"multus-additional-cni-plugins-wlj77\" (UID: \"30488cdf-7b40-4e46-ab07-b94cb7706253\") " pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.142854 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.142805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9xsf\" (UniqueName: \"kubernetes.io/projected/7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae-kube-api-access-j9xsf\") pod \"aws-ebs-csi-driver-node-8jjgr\" (UID: \"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.216448 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.216413    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-nx72m"
Jul 16 17:34:32.224158 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.224137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8s7qq"
Jul 16 17:34:32.228381 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.228355    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2625d2_c2e2_4d64_a7b2_e5def0303940.slice/crio-6a49ccb81ab28bf8673d32bf255ab74868f1384055c36ba7334869cdfabaee0c WatchSource:0}: Error finding container 6a49ccb81ab28bf8673d32bf255ab74868f1384055c36ba7334869cdfabaee0c: Status 404 returned error can't find the container with id 6a49ccb81ab28bf8673d32bf255ab74868f1384055c36ba7334869cdfabaee0c
Jul 16 17:34:32.233699 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.233680    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:32.238164 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.238143    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ljskh"
Jul 16 17:34:32.243819 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.243794    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qzlzg"
Jul 16 17:34:32.244096 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.244065    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d83f116_f57c_42b7_aa3e_06d562f2cc05.slice/crio-a9c7725bb8faf5f70bceacb051f5fcba8a500b0b491fef40d674dea2df720b5e WatchSource:0}: Error finding container a9c7725bb8faf5f70bceacb051f5fcba8a500b0b491fef40d674dea2df720b5e: Status 404 returned error can't find the container with id a9c7725bb8faf5f70bceacb051f5fcba8a500b0b491fef40d674dea2df720b5e
Jul 16 17:34:32.246960 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.246936    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83464c6d_46a8_4fd7_a964_25bdc629e973.slice/crio-3299ecf072f0b04efd6ee0a4063f5141de19b7e01c13db0fb5f7b67437a9e803 WatchSource:0}: Error finding container 3299ecf072f0b04efd6ee0a4063f5141de19b7e01c13db0fb5f7b67437a9e803: Status 404 returned error can't find the container with id 3299ecf072f0b04efd6ee0a4063f5141de19b7e01c13db0fb5f7b67437a9e803
Jul 16 17:34:32.250808 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.250787    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vcz5q"
Jul 16 17:34:32.253268 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.253244    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e969db_03bb_4ef2_8053_4e85ee82b946.slice/crio-522c3a3c26dcd35988e26ba18206e378c3a8d4662f93dfb2effde83abae54807 WatchSource:0}: Error finding container 522c3a3c26dcd35988e26ba18206e378c3a8d4662f93dfb2effde83abae54807: Status 404 returned error can't find the container with id 522c3a3c26dcd35988e26ba18206e378c3a8d4662f93dfb2effde83abae54807
Jul 16 17:34:32.256237 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.256219    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr"
Jul 16 17:34:32.259414 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.259389    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06fccf7d_2fcb_47cf_b0f4_3d4f169a1254.slice/crio-6348dd0c6e07e95272b1e2c091036f90a238cf51edd3dd57c2ce8aeca749dccb WatchSource:0}: Error finding container 6348dd0c6e07e95272b1e2c091036f90a238cf51edd3dd57c2ce8aeca749dccb: Status 404 returned error can't find the container with id 6348dd0c6e07e95272b1e2c091036f90a238cf51edd3dd57c2ce8aeca749dccb
Jul 16 17:34:32.261560 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.261539    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wlj77"
Jul 16 17:34:32.266570 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.266548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:34:32.266933 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.266914    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5d9ea4_04c2_4e7d_bb5c_da249e8fa1ae.slice/crio-1bb129485b5393534ebc479bd694d4340a1ff16db730bcfd3e2ae99cba3d3ff1 WatchSource:0}: Error finding container 1bb129485b5393534ebc479bd694d4340a1ff16db730bcfd3e2ae99cba3d3ff1: Status 404 returned error can't find the container with id 1bb129485b5393534ebc479bd694d4340a1ff16db730bcfd3e2ae99cba3d3ff1
Jul 16 17:34:32.270044 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.270022    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30488cdf_7b40_4e46_ab07_b94cb7706253.slice/crio-5fd95ec43e6bfdec98868de1d5af840c7cd21ead807484dcb7404a0b1732ed28 WatchSource:0}: Error finding container 5fd95ec43e6bfdec98868de1d5af840c7cd21ead807484dcb7404a0b1732ed28: Status 404 returned error can't find the container with id 5fd95ec43e6bfdec98868de1d5af840c7cd21ead807484dcb7404a0b1732ed28
Jul 16 17:34:32.275584 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:32.275564    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393a0ebb_5874_4cec_b039_b84de1526fb4.slice/crio-e948c7ad56b4e7c05cc4b7892a4bbf4f6210633b9b204cae38d956433d3affa3 WatchSource:0}: Error finding container e948c7ad56b4e7c05cc4b7892a4bbf4f6210633b9b204cae38d956433d3affa3: Status 404 returned error can't find the container with id e948c7ad56b4e7c05cc4b7892a4bbf4f6210633b9b204cae38d956433d3affa3
Jul 16 17:34:32.535699 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.535575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:32.535699 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.535612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:32.535699 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.535654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535750    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535753    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535811    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:33.53579241 +0000 UTC m=+3.959522464 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535835    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535862    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535874    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535839    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:33.535831353 +0000 UTC m=+3.959561393 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:32.536043 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:32.535945    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:33.535927176 +0000 UTC m=+3.959657218 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:32.967238 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.967199    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:32 +0000 UTC" deadline="2028-02-06 05:39:34.724303455 +0000 UTC"
Jul 16 17:34:32.967238 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:32.967239    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13668h5m1.757069778s"
Jul 16 17:34:33.095056 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.094905    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"e948c7ad56b4e7c05cc4b7892a4bbf4f6210633b9b204cae38d956433d3affa3"}
Jul 16 17:34:33.097575 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.097539    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vcz5q" event={"ID":"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254","Type":"ContainerStarted","Data":"6348dd0c6e07e95272b1e2c091036f90a238cf51edd3dd57c2ce8aeca749dccb"}
Jul 16 17:34:33.099456 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.099425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljskh" event={"ID":"83464c6d-46a8-4fd7-a964-25bdc629e973","Type":"ContainerStarted","Data":"3299ecf072f0b04efd6ee0a4063f5141de19b7e01c13db0fb5f7b67437a9e803"}
Jul 16 17:34:33.102805 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.102786    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2sbn" event={"ID":"0d83f116-f57c-42b7-aa3e-06d562f2cc05","Type":"ContainerStarted","Data":"a9c7725bb8faf5f70bceacb051f5fcba8a500b0b491fef40d674dea2df720b5e"}
Jul 16 17:34:33.104413 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.104386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nx72m" event={"ID":"5e2625d2-c2e2-4d64-a7b2-e5def0303940","Type":"ContainerStarted","Data":"6a49ccb81ab28bf8673d32bf255ab74868f1384055c36ba7334869cdfabaee0c"}
Jul 16 17:34:33.106066 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.106042    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerStarted","Data":"5fd95ec43e6bfdec98868de1d5af840c7cd21ead807484dcb7404a0b1732ed28"}
Jul 16 17:34:33.107988 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.107963    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr" event={"ID":"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae","Type":"ContainerStarted","Data":"1bb129485b5393534ebc479bd694d4340a1ff16db730bcfd3e2ae99cba3d3ff1"}
Jul 16 17:34:33.109386 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.109361    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzlzg" event={"ID":"f3e969db-03bb-4ef2-8053-4e85ee82b946","Type":"ContainerStarted","Data":"522c3a3c26dcd35988e26ba18206e378c3a8d4662f93dfb2effde83abae54807"}
Jul 16 17:34:33.110727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.110697    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8s7qq" event={"ID":"96ac4f84-129c-48f7-bd5f-0f661b0f80b8","Type":"ContainerStarted","Data":"1f51dcc1cdaa437d167f5c766bc8aa4900af26a923d38613ffd7c362de922c15"}
Jul 16 17:34:33.545098 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.544638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:33.545098 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.544695    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:33.545098 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.544750    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:33.545098 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.544884    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:33.545098 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.544953    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.54493193 +0000 UTC m=+5.968661994 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545336    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545380    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.545365771 +0000 UTC m=+5.969095810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545439    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545458    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545473    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:33.545590 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:33.545500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:35.545492411 +0000 UTC m=+5.969222465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:33.967994 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.967908    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 17:29:32 +0000 UTC" deadline="2028-01-20 19:04:53.139696876 +0000 UTC"
Jul 16 17:34:33.967994 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:33.967949    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13273h30m19.171753605s"
Jul 16 17:34:34.086625 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:34.086591    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:34.086803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:34.086602    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:34.086803 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:34.086748    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:34.086929 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:34.086841    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:34.086929 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:34.086902    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:34.087038 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:34.086969    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:35.572625 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:35.572542    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:35.572625 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:35.572595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:35.572625 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:35.572641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:35.573330 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.572778    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.573330 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.572845    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:39.572823958 +0000 UTC m=+9.996554001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:35.573330 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573282    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.573542 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573341    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:39.57332222 +0000 UTC m=+9.997052277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:35.573542 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573430    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:35.573542 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573446    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:35.573542 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573459    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:35.573542 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:35.573500    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:39.573487345 +0000 UTC m=+9.997217400 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:36.087690 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:36.087656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:36.087858 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:36.087698    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:36.087858 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:36.087656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:36.087858 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:36.087817    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:36.087956 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:36.087905    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:36.088016 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:36.087994    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:38.086531    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:38.086654    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:38.087066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:38.087168    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:38.087223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:38.087337 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:38.087290    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:39.607269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:39.607320    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:39.607360    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607419    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607487    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607497    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:34:47.607481348 +0000 UTC m=+18.031211386 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607538    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:47.607521366 +0000 UTC m=+18.031251417 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607597    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607607    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607616    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:39.607660 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:39.607638    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:34:47.607631065 +0000 UTC m=+18.031361103 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:40.088028 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:40.087946    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:40.088183 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:40.088133    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:40.088400 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:40.088378    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:40.088579 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:40.088551    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:40.088673 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:40.088595    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:40.088731 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:40.088712    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:42.085909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:42.085874    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:42.085909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:42.085887    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:42.086377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:42.085890    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:42.086377 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:42.085996    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:42.086377 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:42.086080    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:42.086377 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:42.086148    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:44.085762 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:44.085515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:44.086202 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:44.085516    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:44.086202 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:44.085555    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:44.086202 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:44.085886    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:44.086202 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:44.085983    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:44.086202 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:44.086081    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:46.085736 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:46.085696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:46.086177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:46.085696    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:46.086177 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:46.085832    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:46.086177 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:46.085884    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:46.086177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:46.085993    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:46.086177 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:46.086070    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:47.669058 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:47.669018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:47.669096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:47.669126    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669199    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669227    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669239    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669257    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669270    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669285    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:03.669260818 +0000 UTC m=+34.092990883 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669305    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:35:03.669295011 +0000 UTC m=+34.093025060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:34:47.669607 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:47.669324    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:03.669314118 +0000 UTC m=+34.093044158 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:34:48.088367 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:48.088333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:48.088558 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:48.088338    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:48.088558 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:48.088470    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:48.088558 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:48.088344    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:48.088714 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:48.088557    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:48.088714 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:48.088642    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:50.086507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:50.086468    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:50.086918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:50.086576    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:50.086918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:50.086610    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:50.086918 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:50.086576    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:50.086918 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:50.086701    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:50.086918 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:50.086795    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:51.886453 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:34:51.886405    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod910cf08b8d7e1547a1739427e869cca2.slice/crio-08960c058016ac788ada838012e848cab3931b0cc13fd30078d99cf79153dc62 WatchSource:0}: Error finding container 08960c058016ac788ada838012e848cab3931b0cc13fd30078d99cf79153dc62: Status 404 returned error can't find the container with id 08960c058016ac788ada838012e848cab3931b0cc13fd30078d99cf79153dc62
Jul 16 17:34:52.086311 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.086269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:52.086909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.086643    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:52.086909 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:52.086759    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:52.087040 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:52.086939    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:52.087040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.087011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:52.090143 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:52.087128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:52.157692 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.157644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"efebe3eec720a164768d4cc8ea53737bf94e5e2cc2ad12b76446347b0dd4cdcb"}
Jul 16 17:34:52.159445 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.159406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ljskh" event={"ID":"83464c6d-46a8-4fd7-a964-25bdc629e973","Type":"ContainerStarted","Data":"f415a2e4d61455b3b1f3d037128659d5f5e5a413001f5f360370994781fb3c6f"}
Jul 16 17:34:52.160544 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.160516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal" event={"ID":"910cf08b8d7e1547a1739427e869cca2","Type":"ContainerStarted","Data":"08960c058016ac788ada838012e848cab3931b0cc13fd30078d99cf79153dc62"}
Jul 16 17:34:52.162764 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.162735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal" event={"ID":"648d6c1142d0e1d3e99bb10c691cfa84","Type":"ContainerStarted","Data":"61ee1b7ece26e698cfda8c5d0721cfd86f054356aa718ca2584510187e1a047d"}
Jul 16 17:34:52.179986 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:52.179943    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ljskh" podStartSLOduration=2.493289711 podStartE2EDuration="22.179928917s" totalImagesPullingTime="19.686639206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:52.179667474 +0000 UTC m=+22.603397533" watchObservedRunningTime="2026-07-16 17:34:52.179928917 +0000 UTC m=+22.603658974"
Jul 16 17:34:53.167683 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.167287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-nx72m" event={"ID":"5e2625d2-c2e2-4d64-a7b2-e5def0303940","Type":"ContainerStarted","Data":"128aeacf7553363e08d93557f4436b1e07412b64f01fc4c170875577bd8aa832"}
Jul 16 17:34:53.169201 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.169157    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="ed6edf13def5bcc7b1ee68af93525d2cc0db701f2cd6d2cbde69dc2c6f281174" exitCode=0
Jul 16 17:34:53.169327 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.169238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"ed6edf13def5bcc7b1ee68af93525d2cc0db701f2cd6d2cbde69dc2c6f281174"}
Jul 16 17:34:53.172295 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.172274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr" event={"ID":"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae","Type":"ContainerStarted","Data":"e765c6c2450c35d7f8cddc3ac303dca8a81ed0472c51f0dc2b41383a7c924e16"}
Jul 16 17:34:53.173857 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.173826    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qzlzg" event={"ID":"f3e969db-03bb-4ef2-8053-4e85ee82b946","Type":"ContainerStarted","Data":"28dd81ff9434c0f4e994285245522692a183f60ca80b4c4ba7d57e072fa3aa3b"}
Jul 16 17:34:53.176613 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.176590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:34:53.177028 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.177003    2576 generic.go:356] "Generic (PLEG): container finished" podID="393a0ebb-5874-4cec-b039-b84de1526fb4" containerID="68c7cd00128a8980c54e9f62f26b30be3bed014b84d16d5d3f0c7aac8e7fd1ec" exitCode=1
Jul 16 17:34:53.177126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.177079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerDied","Data":"68c7cd00128a8980c54e9f62f26b30be3bed014b84d16d5d3f0c7aac8e7fd1ec"}
Jul 16 17:34:53.178919 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.178893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vcz5q" event={"ID":"06fccf7d-2fcb-47cf-b0f4-3d4f169a1254","Type":"ContainerStarted","Data":"8ed88f2850718f8ab71c6c14a7a8bba9e699c91bb20309dc5e2e279aba0025b6"}
Jul 16 17:34:53.181804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.181779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-n2sbn" event={"ID":"0d83f116-f57c-42b7-aa3e-06d562f2cc05","Type":"ContainerStarted","Data":"f533a5a5402bc7604dea81948ca882cae4b070d4abac006ac79d596fc5a3bc16"}
Jul 16 17:34:53.182111 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.182074    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-77.ec2.internal" podStartSLOduration=22.182061676 podStartE2EDuration="22.182061676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:52.195507994 +0000 UTC m=+22.619238055" watchObservedRunningTime="2026-07-16 17:34:53.182061676 +0000 UTC m=+23.605791735"
Jul 16 17:34:53.182306 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.182282    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-nx72m" podStartSLOduration=3.500794679 podStartE2EDuration="23.182277203s" totalImagesPullingTime="19.681482524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.181924123 +0000 UTC m=+23.605654183" watchObservedRunningTime="2026-07-16 17:34:53.182277203 +0000 UTC m=+23.606007264"
Jul 16 17:34:53.196995 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.196954    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qzlzg" podStartSLOduration=3.517929864 podStartE2EDuration="23.196938746s" totalImagesPullingTime="19.679008882s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.196675925 +0000 UTC m=+23.620405985" watchObservedRunningTime="2026-07-16 17:34:53.196938746 +0000 UTC m=+23.620668807"
Jul 16 17:34:53.212544 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.212500    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vcz5q" podStartSLOduration=3.237732229 podStartE2EDuration="23.212483369s" totalImagesPullingTime="19.97475114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.212416207 +0000 UTC m=+23.636146267" watchObservedRunningTime="2026-07-16 17:34:53.212483369 +0000 UTC m=+23.636213434"
Jul 16 17:34:53.481029 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.480920    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:53.481590 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.481568    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:53.497494 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:53.497428    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-n2sbn" podStartSLOduration=3.846091431 podStartE2EDuration="23.497409761s" totalImagesPullingTime="19.65131833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:53.252772647 +0000 UTC m=+23.676502706" watchObservedRunningTime="2026-07-16 17:34:53.497409761 +0000 UTC m=+23.921139821"
Jul 16 17:34:54.085601 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.085573    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:54.085727 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:54.085711    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:54.086120 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.086097    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:54.086216 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:54.086195    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:54.086279 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.086261    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:54.086343 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:54.086325    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:54.186649 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.186616    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8s7qq" event={"ID":"96ac4f84-129c-48f7-bd5f-0f661b0f80b8","Type":"ContainerStarted","Data":"f633ec74263b6779e26c1fbdde6d06ce0c0d14749f46a4f6e64471d264765f03"}
Jul 16 17:34:54.189397 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.189332    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:34:54.192507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.190853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"4c2cf019309edfe65a4525bfbd2c6be63dfe2f185091aece838b94812326662e"}
Jul 16 17:34:54.192507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.191933    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:54.192507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.192076    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-n2sbn"
Jul 16 17:34:54.203928 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.203848    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8s7qq" podStartSLOduration=17.382904006 podStartE2EDuration="24.203829325s" totalImagesPullingTime="6.820925319s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:54.203597378 +0000 UTC m=+24.627327438" watchObservedRunningTime="2026-07-16 17:34:54.203829325 +0000 UTC m=+24.627559385"
Jul 16 17:34:54.913201 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.912776    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 17:34:54.988099 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.988004    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T17:34:54.912798258Z","UUID":"45ef7c4b-1e29-4555-9573-95fee8798580","Handler":null,"Name":"","Endpoint":""}
Jul 16 17:34:54.989657 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.989636    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 17:34:54.989774 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:54.989689    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 17:34:55.195475 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.195375    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr" event={"ID":"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae","Type":"ContainerStarted","Data":"977d0268b868b96a28240fb1879b9c872db403ecfef280688ab58dda320f3887"}
Jul 16 17:34:55.199471 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.199427    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:34:55.199933 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.199900    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"1e1d174124f75055f5a07ab2d7937a766028e99ca8e6067f411a47314d2056be"}
Jul 16 17:34:55.200024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.200001    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"bab9f1273fb8185f768dc0984954b1b58eb36409d49f28bd484682a8eb381e2d"}
Jul 16 17:34:55.200081 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.200056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"7d17f5d69355a159604737049ad28bd363c5b794d74a8518e43c5ab11f834acb"}
Jul 16 17:34:55.201614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.201590    2576 generic.go:356] "Generic (PLEG): container finished" podID="910cf08b8d7e1547a1739427e869cca2" containerID="5e5b0d5ef8cd5b6bf5b2e3962ed748769c7400ff1e9be5a96564591b5cc63f51" exitCode=0
Jul 16 17:34:55.201715 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:55.201624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal" event={"ID":"910cf08b8d7e1547a1739427e869cca2","Type":"ContainerDied","Data":"5e5b0d5ef8cd5b6bf5b2e3962ed748769c7400ff1e9be5a96564591b5cc63f51"}
Jul 16 17:34:56.086024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:56.085990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:56.086024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:56.086017    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:56.086237 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:56.085990    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:56.086237 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:56.086095    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:56.086237 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:56.086173    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:56.086340 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:56.086255    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:56.206183 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:56.206146    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal" event={"ID":"910cf08b8d7e1547a1739427e869cca2","Type":"ContainerStarted","Data":"6c479a6ffdeff5658bc678edc36c44f485d7850973019ba57f18dd32a260e758"}
Jul 16 17:34:56.220918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:56.220872    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-77.ec2.internal" podStartSLOduration=25.220852771 podStartE2EDuration="25.220852771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:56.220702645 +0000 UTC m=+26.644432708" watchObservedRunningTime="2026-07-16 17:34:56.220852771 +0000 UTC m=+26.644582834"
Jul 16 17:34:58.085976 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.085807    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:34:58.086534 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.085815    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:34:58.086534 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:58.086058    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:34:58.086534 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.085820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:34:58.086534 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:58.086130    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:34:58.086534 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:34:58.086200    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:34:58.213538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.213506    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:34:58.213926 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.213902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"fc906ceba56d6cc10bfde900f622f510be925da997ba4180542f3a75fd68901b"}
Jul 16 17:34:58.215505 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.215482    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="8bcc7c785377bb1e90f787cbc590385b1c8adf3fa5eb3edb0de983dc42abea79" exitCode=0
Jul 16 17:34:58.215603 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.215552    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"8bcc7c785377bb1e90f787cbc590385b1c8adf3fa5eb3edb0de983dc42abea79"}
Jul 16 17:34:58.217687 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.217663    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr" event={"ID":"7f5d9ea4-04c2-4e7d-bb5c-da249e8fa1ae","Type":"ContainerStarted","Data":"71f1295d49bf4b05b66650ec08dc800f93abecef2700eabe9761b9bcc7a532e8"}
Jul 16 17:34:58.262512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:34:58.262457    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8jjgr" podStartSLOduration=3.38229062 podStartE2EDuration="28.262423337s" totalImagesPullingTime="24.880132717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:34:58.262314003 +0000 UTC m=+28.686044062" watchObservedRunningTime="2026-07-16 17:34:58.262423337 +0000 UTC m=+28.686153397"
Jul 16 17:35:00.088700 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.088568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:00.089270 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.088632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:00.089270 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:00.088764    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:35:00.089270 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:00.088859    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:35:00.089270 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.088632    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:00.089270 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:00.088979    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:35:00.225041 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.224968    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:35:00.225338 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.225314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"0faf56953ecb3df3c53f997170eafbcaa71126dd1df68cc54d03f8a625be4ccf"}
Jul 16 17:35:00.225617 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.225592    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:00.226593 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.225652    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:00.226593 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.225844    2576 scope.go:117] "RemoveContainer" containerID="68c7cd00128a8980c54e9f62f26b30be3bed014b84d16d5d3f0c7aac8e7fd1ec"
Jul 16 17:35:00.242817 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:00.242798    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:01.233040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.232956    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:35:01.233535 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.233499    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" event={"ID":"393a0ebb-5874-4cec-b039-b84de1526fb4","Type":"ContainerStarted","Data":"29169eae4686ad06b5b1a6c4c99e0e550fddd4e7193b5fe68d42b4a31dc0352c"}
Jul 16 17:35:01.234751 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.234727    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:01.238902 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.238801    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="4764923dccdbea7701bf544c4e0e2a1df35a639e561cd78ca5611c63183cca1b" exitCode=0
Jul 16 17:35:01.238902 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.238838    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"4764923dccdbea7701bf544c4e0e2a1df35a639e561cd78ca5611c63183cca1b"}
Jul 16 17:35:01.252832 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.252805    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:01.276589 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.276537    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs" podStartSLOduration=9.780462115 podStartE2EDuration="31.276520968s" totalImagesPullingTime="21.496058853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:01.276140248 +0000 UTC m=+31.699870327" watchObservedRunningTime="2026-07-16 17:35:01.276520968 +0000 UTC m=+31.700251031"
Jul 16 17:35:01.951361 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.951331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48hrx"]
Jul 16 17:35:01.951544 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.951508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:01.951656 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:01.951632    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:35:01.954501 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.954475    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fqbvg"]
Jul 16 17:35:01.954628 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.954582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:01.954697 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:01.954675    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:35:01.955318 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.955296    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4z97z"]
Jul 16 17:35:01.955426 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:01.955388    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:01.955501 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:01.955484    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:35:03.086185 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.086006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:03.086608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.086043    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:03.086608 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.086261    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:35:03.086608 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.086396    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:35:03.245151 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.245077    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="d9bb91e98fe7f32438d7dda06629207353e8a228b03833fa8689a77fad6d3e0d" exitCode=0
Jul 16 17:35:03.245282 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.245164    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"d9bb91e98fe7f32438d7dda06629207353e8a228b03833fa8689a77fad6d3e0d"}
Jul 16 17:35:03.690166 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.690135    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:03.690166 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.690168    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:03.690193    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690276    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690277    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690295    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690318    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret podName:854d98e3-464a-4550-b809-e0964d0c8175 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:35.690309327 +0000 UTC m=+66.114039366 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret") pod "global-pull-secret-syncer-4z97z" (UID: "854d98e3-464a-4550-b809-e0964d0c8175") : object "kube-system"/"original-pull-secret" not registered
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690332    2576 projected.go:196] Error preparing data for projected volume kube-api-access-nhc76 for pod openshift-network-diagnostics/network-check-target-fqbvg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:03.690336 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690335    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs podName:f861f763-4c0a-4bfa-9880-aabe7948600b nodeName:}" failed. No retries permitted until 2026-07-16 17:35:35.690328952 +0000 UTC m=+66.114058990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs") pod "network-metrics-daemon-48hrx" (UID: "f861f763-4c0a-4bfa-9880-aabe7948600b") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 17:35:03.690605 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:03.690385    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76 podName:bb52c6e9-2de3-410b-ade0-87c4631fa276 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:35.690367855 +0000 UTC m=+66.114097910 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nhc76" (UniqueName: "kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76") pod "network-check-target-fqbvg" (UID: "bb52c6e9-2de3-410b-ade0-87c4631fa276") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 17:35:04.085650 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:04.085622    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:04.085796 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:04.085731    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fqbvg" podUID="bb52c6e9-2de3-410b-ade0-87c4631fa276"
Jul 16 17:35:05.085595 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.085558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:05.086063 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.085559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:05.086063 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:05.085703    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48hrx" podUID="f861f763-4c0a-4bfa-9880-aabe7948600b"
Jul 16 17:35:05.086063 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:05.085828    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4z97z" podUID="854d98e3-464a-4550-b809-e0964d0c8175"
Jul 16 17:35:05.915853 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.915779    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-77.ec2.internal" event="NodeReady"
Jul 16 17:35:05.916020 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.915924    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 17:35:05.959258 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.959224    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z4bnd"]
Jul 16 17:35:05.982450 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.982396    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8rfjs"]
Jul 16 17:35:05.982605 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.982532    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:05.984890 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.984868    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 17:35:05.985023 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.984968    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 17:35:05.985084 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.985075    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jn8p6\""
Jul 16 17:35:05.998403 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.998382    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rfjs"]
Jul 16 17:35:05.998403 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.998408    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4bnd"]
Jul 16 17:35:05.998580 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:05.998517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.001009 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.000988    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 17:35:06.001140 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.000988    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-z92qr\""
Jul 16 17:35:06.001140 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.001033    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 17:35:06.001140 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.001077    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 17:35:06.085917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.085883    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:06.088857 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.088828    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5khsp\""
Jul 16 17:35:06.088857 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.088826    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:06.089050 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.088827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:06.107883 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.107862    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c796012-9277-4e77-8226-60fd5be1dbd8-cert\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.108001 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.107887    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.108001 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.107905    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcfnq\" (UniqueName: \"kubernetes.io/projected/6c796012-9277-4e77-8226-60fd5be1dbd8-kube-api-access-kcfnq\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.108001 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.107923    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7956263e-465c-4266-b46d-e271490872ed-tmp-dir\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.108133 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.107997    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7956263e-465c-4266-b46d-e271490872ed-config-volume\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.108133 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.108021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/7956263e-465c-4266-b46d-e271490872ed-kube-api-access-bq6pc\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.208826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c796012-9277-4e77-8226-60fd5be1dbd8-cert\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.208826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208792    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.208826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcfnq\" (UniqueName: \"kubernetes.io/projected/6c796012-9277-4e77-8226-60fd5be1dbd8-kube-api-access-kcfnq\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.208826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208831    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7956263e-465c-4266-b46d-e271490872ed-tmp-dir\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.209121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7956263e-465c-4266-b46d-e271490872ed-config-volume\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.209121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.208871    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/7956263e-465c-4266-b46d-e271490872ed-kube-api-access-bq6pc\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.209121 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:06.208932    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:06.209121 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:06.209013    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:06.708993185 +0000 UTC m=+37.132723244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:06.209303 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.209174    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7956263e-465c-4266-b46d-e271490872ed-tmp-dir\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.209391 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.209372    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7956263e-465c-4266-b46d-e271490872ed-config-volume\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.213373 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.213344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c796012-9277-4e77-8226-60fd5be1dbd8-cert\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.218660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.218639    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcfnq\" (UniqueName: \"kubernetes.io/projected/6c796012-9277-4e77-8226-60fd5be1dbd8-kube-api-access-kcfnq\") pod \"ingress-canary-8rfjs\" (UID: \"6c796012-9277-4e77-8226-60fd5be1dbd8\") " pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.219130 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.219102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq6pc\" (UniqueName: \"kubernetes.io/projected/7956263e-465c-4266-b46d-e271490872ed-kube-api-access-bq6pc\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.309885 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.309844    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8rfjs"
Jul 16 17:35:06.713861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:06.713825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:06.714047 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:06.713975    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:06.714047 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:06.714042    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:07.714020656 +0000 UTC m=+38.137750712 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:07.086309 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.086267    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:07.086309 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.086298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:07.089311 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.089288    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:07.089311 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.089307    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxx5g\""
Jul 16 17:35:07.089527 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.089330    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:07.720911 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.720882    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:07.721056 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:07.721037    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:07.721100 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:07.721094    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:09.721077942 +0000 UTC m=+40.144807981 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:07.919852 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:07.919639    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8rfjs"]
Jul 16 17:35:07.924370 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:07.924344    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c796012_9277_4e77_8226_60fd5be1dbd8.slice/crio-d88b4a71a33f365f8a3351607a75a0efa202e61998692c7d68027dc5f0ebdadc WatchSource:0}: Error finding container d88b4a71a33f365f8a3351607a75a0efa202e61998692c7d68027dc5f0ebdadc: Status 404 returned error can't find the container with id d88b4a71a33f365f8a3351607a75a0efa202e61998692c7d68027dc5f0ebdadc
Jul 16 17:35:08.258117 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:08.258039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rfjs" event={"ID":"6c796012-9277-4e77-8226-60fd5be1dbd8","Type":"ContainerStarted","Data":"d88b4a71a33f365f8a3351607a75a0efa202e61998692c7d68027dc5f0ebdadc"}
Jul 16 17:35:08.260534 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:08.260506    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="3efb11e736ff2712f5f23c8efadcaea35b5a15517f99e63bbbfb945752825686" exitCode=0
Jul 16 17:35:08.260644 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:08.260573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"3efb11e736ff2712f5f23c8efadcaea35b5a15517f99e63bbbfb945752825686"}
Jul 16 17:35:09.265291 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:09.265255    2576 generic.go:356] "Generic (PLEG): container finished" podID="30488cdf-7b40-4e46-ab07-b94cb7706253" containerID="8516a0b177f252fc1d399cc1e6e970815403699482fdf9552e92eb52ea43d776" exitCode=0
Jul 16 17:35:09.265719 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:09.265322    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerDied","Data":"8516a0b177f252fc1d399cc1e6e970815403699482fdf9552e92eb52ea43d776"}
Jul 16 17:35:09.735088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:09.735050    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:09.735252 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:09.735168    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:09.735252 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:09.735229    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:13.7352154 +0000 UTC m=+44.158945438 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:10.271476 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:10.271419    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wlj77" event={"ID":"30488cdf-7b40-4e46-ab07-b94cb7706253","Type":"ContainerStarted","Data":"acee57bb906c9ae3261be664ceb87078d2c7157c77aa9a209ffc3758d55ca4b7"}
Jul 16 17:35:10.296709 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:10.295909    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wlj77" podStartSLOduration=15.875876942 podStartE2EDuration="40.295888482s" totalImagesPullingTime="24.42001154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:10.294496052 +0000 UTC m=+40.718226103" watchObservedRunningTime="2026-07-16 17:35:10.295888482 +0000 UTC m=+40.719618547"
Jul 16 17:35:12.277348 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:12.277259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8rfjs" event={"ID":"6c796012-9277-4e77-8226-60fd5be1dbd8","Type":"ContainerStarted","Data":"c39bf6df6de9bc2ff1de587a6268502f5c35457471c45c4b69d530577528dadd"}
Jul 16 17:35:12.294953 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:12.294901    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8rfjs" podStartSLOduration=3.365762129 podStartE2EDuration="7.294886844s" totalImagesPullingTime="3.929124715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:12.294323504 +0000 UTC m=+42.718053567" watchObservedRunningTime="2026-07-16 17:35:12.294886844 +0000 UTC m=+42.718616932"
Jul 16 17:35:13.765047 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:13.765005    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:13.765558 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:13.765169    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:13.765558 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:13.765265    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.765245119 +0000 UTC m=+52.188975174 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:17.725290 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.725259    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7mght"]
Jul 16 17:35:17.765176 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.765144    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-b8db8bc8d-dnsc4"]
Jul 16 17:35:17.765319 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.765276    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.767538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.767514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 17:35:17.767677 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.767643    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 17:35:17.767926 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.767899    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 17:35:17.768041 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.767951    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2lnp5\""
Jul 16 17:35:17.768041 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.767967    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 17:35:17.775583 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.775563    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 17:35:17.777938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.777922    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7mght"]
Jul 16 17:35:17.778009 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.777949    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b8db8bc8d-dnsc4"]
Jul 16 17:35:17.778040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.778016    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.780322 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780291    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 16 17:35:17.780428 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780411    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 16 17:35:17.780568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780544    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 16 17:35:17.780681 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780570    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jknlw\""
Jul 16 17:35:17.780852 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780832    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 17:35:17.780924 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780885    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 17:35:17.780979 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.780961    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 16 17:35:17.793125 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793105    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-tmp\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793223 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793130    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-snapshots\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793223 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793149    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793223 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.793346 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbcd895-ffcb-4eb3-be2b-871643f6110b-serving-cert\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793346 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793295    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-default-certificate\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.793346 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793346 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs58v\" (UniqueName: \"kubernetes.io/projected/3dbcd895-ffcb-4eb3-be2b-871643f6110b-kube-api-access-xs58v\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.793520 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.793520 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793376    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-stats-auth\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.793520 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.793423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfzrm\" (UniqueName: \"kubernetes.io/projected/90995984-c754-40fd-acb4-e505ab467da1-kube-api-access-hfzrm\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.832803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.832771    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"]
Jul 16 17:35:17.854642 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.854566    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:35:17.854809 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.854685    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"
Jul 16 17:35:17.857659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.857630    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hwmh8\""
Jul 16 17:35:17.859285 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.859260    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 17:35:17.859410 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.859268    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 17:35:17.870599 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.870578    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"]
Jul 16 17:35:17.870701 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.870609    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:35:17.870701 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.870684    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.873236 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.873217    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 17:35:17.873339 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.873240    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 17:35:17.873339 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.873292    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 17:35:17.873492 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.873367    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qdqxv\""
Jul 16 17:35:17.878828 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.878808    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 17:35:17.894584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-default-certificate\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.894686 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894598    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.894686 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.894854 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.894885 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894873    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.894917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894901    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdnwx\" (UniqueName: \"kubernetes.io/projected/dbd576ff-bc74-4b42-ad1a-1610928fb9f3-kube-api-access-kdnwx\") pod \"volume-data-source-validator-78495597f6-zck5v\" (UID: \"dbd576ff-bc74-4b42-ad1a-1610928fb9f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"
Jul 16 17:35:17.894949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894934    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs58v\" (UniqueName: \"kubernetes.io/projected/3dbcd895-ffcb-4eb3-be2b-871643f6110b-kube-api-access-xs58v\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.894985 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.894962    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.895024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-stats-auth\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.895060 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.895048    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:17.895098 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895066    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.895143 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895100    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfzrm\" (UniqueName: \"kubernetes.io/projected/90995984-c754-40fd-acb4-e505ab467da1-kube-api-access-hfzrm\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.895143 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.895118    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:18.395098094 +0000 UTC m=+48.818828153 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : secret "router-metrics-certs-default" not found
Jul 16 17:35:17.895249 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.895249 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895184    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-tmp\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895249 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-snapshots\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895248    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69w94\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.895377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.895377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895366    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbcd895-ffcb-4eb3-be2b-871643f6110b-serving-cert\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895400    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.895608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895423    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.895608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-tmp\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.895608 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.895554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:18.395535424 +0000 UTC m=+48.819265492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:17.895837 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.895818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/3dbcd895-ffcb-4eb3-be2b-871643f6110b-snapshots\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.896129 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.896105    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3dbcd895-ffcb-4eb3-be2b-871643f6110b-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.899003 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.898979    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3dbcd895-ffcb-4eb3-be2b-871643f6110b-serving-cert\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.899079 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.899060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-default-certificate\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.899197 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.899175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-stats-auth\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.911473 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.911427    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs58v\" (UniqueName: \"kubernetes.io/projected/3dbcd895-ffcb-4eb3-be2b-871643f6110b-kube-api-access-xs58v\") pod \"insights-operator-68d9dd94cf-7mght\" (UID: \"3dbcd895-ffcb-4eb3-be2b-871643f6110b\") " pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:17.918486 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.918457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfzrm\" (UniqueName: \"kubernetes.io/projected/90995984-c754-40fd-acb4-e505ab467da1-kube-api-access-hfzrm\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:17.996698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996606    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.996698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996690    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.996914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996717    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.996914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996737    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.996914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996761    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdnwx\" (UniqueName: \"kubernetes.io/projected/dbd576ff-bc74-4b42-ad1a-1610928fb9f3-kube-api-access-kdnwx\") pod \"volume-data-source-validator-78495597f6-zck5v\" (UID: \"dbd576ff-bc74-4b42-ad1a-1610928fb9f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"
Jul 16 17:35:17.996914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996829    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.996914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.996860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.997146 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.997052    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.997146 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.997078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69w94\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.997251 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.997148    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.997302 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.997251    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:17.997302 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.997264    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-548664f95f-jxnsj: secret "image-registry-tls" not found
Jul 16 17:35:17.997372 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.997313    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.997372 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:17.997322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls podName:05c6e21e-4d56-437d-a651-c35b2be32e4e nodeName:}" failed. No retries permitted until 2026-07-16 17:35:18.497304589 +0000 UTC m=+48.921034643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls") pod "image-registry-548664f95f-jxnsj" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e") : secret "image-registry-tls" not found
Jul 16 17:35:17.997976 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.997953    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.999355 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.999330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:17.999355 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:17.999347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:18.011504 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.011473    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:18.011870 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.011854    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69w94\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:18.011949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.011922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdnwx\" (UniqueName: \"kubernetes.io/projected/dbd576ff-bc74-4b42-ad1a-1610928fb9f3-kube-api-access-kdnwx\") pod \"volume-data-source-validator-78495597f6-zck5v\" (UID: \"dbd576ff-bc74-4b42-ad1a-1610928fb9f3\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"
Jul 16 17:35:18.075623 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.075590    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7mght"
Jul 16 17:35:18.175458 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.174548    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"
Jul 16 17:35:18.251001 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.250946    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7mght"]
Jul 16 17:35:18.254346 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:18.254194    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbcd895_ffcb_4eb3_be2b_871643f6110b.slice/crio-6284b61b941d49f704bbed735791fd37dec15285e08bc3c7f6ce78def847f04d WatchSource:0}: Error finding container 6284b61b941d49f704bbed735791fd37dec15285e08bc3c7f6ce78def847f04d: Status 404 returned error can't find the container with id 6284b61b941d49f704bbed735791fd37dec15285e08bc3c7f6ce78def847f04d
Jul 16 17:35:18.291828 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.291766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7mght" event={"ID":"3dbcd895-ffcb-4eb3-be2b-871643f6110b","Type":"ContainerStarted","Data":"6284b61b941d49f704bbed735791fd37dec15285e08bc3c7f6ce78def847f04d"}
Jul 16 17:35:18.331116 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.331088    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v"]
Jul 16 17:35:18.334021 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:18.333996    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbd576ff_bc74_4b42_ad1a_1610928fb9f3.slice/crio-ea5944560a036ff3d0675c55cb902989a0925e9e237f80671e3ae6fa8be565d1 WatchSource:0}: Error finding container ea5944560a036ff3d0675c55cb902989a0925e9e237f80671e3ae6fa8be565d1: Status 404 returned error can't find the container with id ea5944560a036ff3d0675c55cb902989a0925e9e237f80671e3ae6fa8be565d1
Jul 16 17:35:18.400679 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.400644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:18.400856 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.400713    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:18.400856 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.400795    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:18.400856 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.400852    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.400839467 +0000 UTC m=+49.824569505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:18.400977 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.400868    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.400861463 +0000 UTC m=+49.824591501 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : secret "router-metrics-certs-default" not found
Jul 16 17:35:18.501402 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:18.501321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:18.501570 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.501453    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:18.501570 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.501468    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-548664f95f-jxnsj: secret "image-registry-tls" not found
Jul 16 17:35:18.501570 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:18.501529    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls podName:05c6e21e-4d56-437d-a651-c35b2be32e4e nodeName:}" failed. No retries permitted until 2026-07-16 17:35:19.501505722 +0000 UTC m=+49.925235760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls") pod "image-registry-548664f95f-jxnsj" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e") : secret "image-registry-tls" not found
Jul 16 17:35:19.295867 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:19.295827    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v" event={"ID":"dbd576ff-bc74-4b42-ad1a-1610928fb9f3","Type":"ContainerStarted","Data":"ea5944560a036ff3d0675c55cb902989a0925e9e237f80671e3ae6fa8be565d1"}
Jul 16 17:35:19.408666 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:19.408628    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:19.408831 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:19.408726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:19.408831 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.408799    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.408775562 +0000 UTC m=+51.832505623 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:19.408934 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.408871    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:19.408934 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.408931    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.408915871 +0000 UTC m=+51.832645927 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : secret "router-metrics-certs-default" not found
Jul 16 17:35:19.509347 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:19.509309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:19.509524 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.509474    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:19.509524 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.509492    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-548664f95f-jxnsj: secret "image-registry-tls" not found
Jul 16 17:35:19.509609 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:19.509554    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls podName:05c6e21e-4d56-437d-a651-c35b2be32e4e nodeName:}" failed. No retries permitted until 2026-07-16 17:35:21.509539753 +0000 UTC m=+51.933269809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls") pod "image-registry-548664f95f-jxnsj" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e") : secret "image-registry-tls" not found
Jul 16 17:35:20.299336 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:20.299305    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v" event={"ID":"dbd576ff-bc74-4b42-ad1a-1610928fb9f3","Type":"ContainerStarted","Data":"9cb87080e5863f1d230893c13906187e304583370a7a03bfa60f09caf0682e43"}
Jul 16 17:35:20.318881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:20.318838    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-zck5v" podStartSLOduration=1.50938642 podStartE2EDuration="3.318822074s" totalImagesPullingTime="1.809435654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:20.318272965 +0000 UTC m=+50.742003025" watchObservedRunningTime="2026-07-16 17:35:20.318822074 +0000 UTC m=+50.742552130"
Jul 16 17:35:21.303679 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.303643    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7mght" event={"ID":"3dbcd895-ffcb-4eb3-be2b-871643f6110b","Type":"ContainerStarted","Data":"e616064095f19b0b31e18adcb19233d02e5cfde2c839e3c9dfff5b0082bdf8cb"}
Jul 16 17:35:21.321486 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.321414    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-7mght" podStartSLOduration=2.1149413519999998 podStartE2EDuration="4.321393198s" totalImagesPullingTime="2.206451846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:21.320791145 +0000 UTC m=+51.744521204" watchObservedRunningTime="2026-07-16 17:35:21.321393198 +0000 UTC m=+51.745123260"
Jul 16 17:35:21.423950 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.423906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:21.424119 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.423997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:21.424119 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.424096    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.424075661 +0000 UTC m=+55.847805716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:21.424237 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.424125    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:21.424237 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.424185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.424168585 +0000 UTC m=+55.847898640 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : secret "router-metrics-certs-default" not found
Jul 16 17:35:21.525279 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.525244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:21.525484 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.525412    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:21.525484 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.525453    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-548664f95f-jxnsj: secret "image-registry-tls" not found
Jul 16 17:35:21.525615 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.525528    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls podName:05c6e21e-4d56-437d-a651-c35b2be32e4e nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.52549983 +0000 UTC m=+55.949229885 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls") pod "image-registry-548664f95f-jxnsj" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e") : secret "image-registry-tls" not found
Jul 16 17:35:21.823840 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.823802    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-ts6vm"]
Jul 16 17:35:21.827052 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.827024    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:21.827206 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.827170    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 17:35:21.827268 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:21.827228    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls podName:7956263e-465c-4266-b46d-e271490872ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.827214447 +0000 UTC m=+68.250944488 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls") pod "dns-default-z4bnd" (UID: "7956263e-465c-4266-b46d-e271490872ed") : secret "dns-default-metrics-tls" not found
Jul 16 17:35:21.843941 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.843914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:21.844134 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.844111    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts6vm"]
Jul 16 17:35:21.846639 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.846610    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 17:35:21.847577 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.847558    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 17:35:21.847886 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.847864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dpvg9\""
Jul 16 17:35:21.928394 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.928361    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:21.928394 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.928394    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-data-volume\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:21.928647 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.928447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc877\" (UniqueName: \"kubernetes.io/projected/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-api-access-vc877\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:21.928647 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.928469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:21.928647 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:21.928536    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-crio-socket\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.029763 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.029727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc877\" (UniqueName: \"kubernetes.io/projected/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-api-access-vc877\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.029763 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.029768    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.029949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.029795    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-crio-socket\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.029949 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.029886    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:22.029949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.029919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.030082 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.029952    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.529936085 +0000 UTC m=+52.953666122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:22.030082 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.029972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-data-volume\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.030082 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.029980    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:22.030082 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.030008    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-crio-socket\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.030082 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.030026    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:22.530011309 +0000 UTC m=+52.953741377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:22.030292 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.030275    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-data-volume\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.042943 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.042922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc877\" (UniqueName: \"kubernetes.io/projected/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-api-access-vc877\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.533347 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.533311    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.533793 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:22.533370    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:22.533793 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.533460    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:22.533793 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.533488    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:22.533793 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.533534    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.533518675 +0000 UTC m=+53.957248725 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:22.533793 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:22.533557    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:23.533544596 +0000 UTC m=+53.957274633 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:23.540962 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:23.540906    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:23.541403 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:23.541029    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:23.541403 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:23.541086    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:23.541403 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:23.541105    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:23.541403 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:23.541157    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.541143902 +0000 UTC m=+55.964873940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:23.541403 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:23.541170    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:25.54116463 +0000 UTC m=+55.964894667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:23.813826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:23.813804    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzlzg_f3e969db-03bb-4ef2-8053-4e85ee82b946/dns-node-resolver/0.log"
Jul 16 17:35:24.813574 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:24.813547    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nx72m_5e2625d2-c2e2-4d64-a7b2-e5def0303940/node-ca/0.log"
Jul 16 17:35:25.414031 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.414002    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rfjs_6c796012-9277-4e77-8226-60fd5be1dbd8/serve-healthcheck-canary/0.log"
Jul 16 17:35:25.458949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.455966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:25.458949 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.456184    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:33.456155366 +0000 UTC m=+63.879885406 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : configmap references non-existent config key: service-ca.crt
Jul 16 17:35:25.458949 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.456234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:25.458949 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.456455    2576 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 16 17:35:25.458949 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.456506    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs podName:90995984-c754-40fd-acb4-e505ab467da1 nodeName:}" failed. No retries permitted until 2026-07-16 17:35:33.456492033 +0000 UTC m=+63.880222084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs") pod "router-default-b8db8bc8d-dnsc4" (UID: "90995984-c754-40fd-acb4-e505ab467da1") : secret "router-metrics-certs-default" not found
Jul 16 17:35:25.557741 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.557708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:25.557925 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.557759    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:25.557925 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:25.557815    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:25.557925 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557893    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:25.557925 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557904    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:25.558102 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557945    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 17:35:25.558102 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557956    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.557943049 +0000 UTC m=+59.981673100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:25.558102 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557961    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-548664f95f-jxnsj: secret "image-registry-tls" not found
Jul 16 17:35:25.558102 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.557980    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:29.557961103 +0000 UTC m=+59.981691163 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:25.558102 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:25.558009    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls podName:05c6e21e-4d56-437d-a651-c35b2be32e4e nodeName:}" failed. No retries permitted until 2026-07-16 17:35:33.557993741 +0000 UTC m=+63.981723803 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls") pod "image-registry-548664f95f-jxnsj" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e") : secret "image-registry-tls" not found
Jul 16 17:35:29.587039 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:29.586990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:29.587580 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:29.587078    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:29.587580 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:29.587123    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 17:35:29.587580 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:29.587206    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.587186292 +0000 UTC m=+68.010916343 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : configmap "kube-rbac-proxy" not found
Jul 16 17:35:29.587580 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:29.587224    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:29.587580 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:35:29.587261    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls podName:b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed nodeName:}" failed. No retries permitted until 2026-07-16 17:35:37.587250736 +0000 UTC m=+68.010980798 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls") pod "insights-runtime-extractor-ts6vm" (UID: "b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed") : secret "insights-runtime-extractor-tls" not found
Jul 16 17:35:33.255703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.255664    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ktrzs"
Jul 16 17:35:33.519245 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.519157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:33.519397 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.519256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:33.519791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.519764    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90995984-c754-40fd-acb4-e505ab467da1-service-ca-bundle\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:33.521631 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.521606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90995984-c754-40fd-acb4-e505ab467da1-metrics-certs\") pod \"router-default-b8db8bc8d-dnsc4\" (UID: \"90995984-c754-40fd-acb4-e505ab467da1\") " pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:33.620076 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.620041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:33.622464 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.622423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"image-registry-548664f95f-jxnsj\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") " pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:33.688833 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.688806    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-jknlw\""
Jul 16 17:35:33.696936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.696918    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:33.783667 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.783643    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qdqxv\""
Jul 16 17:35:33.791187 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.791025    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:33.827936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.827851    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-b8db8bc8d-dnsc4"]
Jul 16 17:35:33.830401 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:33.830374    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90995984_c754_40fd_acb4_e505ab467da1.slice/crio-60380e5cbe3e296a7bdc33bd6f27e80d7f17f4ed59de741705dac5a718096d67 WatchSource:0}: Error finding container 60380e5cbe3e296a7bdc33bd6f27e80d7f17f4ed59de741705dac5a718096d67: Status 404 returned error can't find the container with id 60380e5cbe3e296a7bdc33bd6f27e80d7f17f4ed59de741705dac5a718096d67
Jul 16 17:35:33.931353 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:33.931321    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:35:33.933986 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:33.933962    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c6e21e_4d56_437d_a651_c35b2be32e4e.slice/crio-969af2cd5fc0ceb74774117d123d38ea0126e4e55ece50d1d48e3528ceeae6b0 WatchSource:0}: Error finding container 969af2cd5fc0ceb74774117d123d38ea0126e4e55ece50d1d48e3528ceeae6b0: Status 404 returned error can't find the container with id 969af2cd5fc0ceb74774117d123d38ea0126e4e55ece50d1d48e3528ceeae6b0
Jul 16 17:35:34.337811 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.337780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" event={"ID":"05c6e21e-4d56-437d-a651-c35b2be32e4e","Type":"ContainerStarted","Data":"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"}
Jul 16 17:35:34.338268 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.337836    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" event={"ID":"05c6e21e-4d56-437d-a651-c35b2be32e4e","Type":"ContainerStarted","Data":"969af2cd5fc0ceb74774117d123d38ea0126e4e55ece50d1d48e3528ceeae6b0"}
Jul 16 17:35:34.338268 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.337890    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:35:34.339420 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.339392    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4" event={"ID":"90995984-c754-40fd-acb4-e505ab467da1","Type":"ContainerStarted","Data":"c371309764f98590f0774f76936527f0d2a3fdcf8a589a4d74643e8d785381e6"}
Jul 16 17:35:34.339563 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.339460    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4" event={"ID":"90995984-c754-40fd-acb4-e505ab467da1","Type":"ContainerStarted","Data":"60380e5cbe3e296a7bdc33bd6f27e80d7f17f4ed59de741705dac5a718096d67"}
Jul 16 17:35:34.359641 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.359596    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" podStartSLOduration=17.359582644 podStartE2EDuration="17.359582644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:34.358760235 +0000 UTC m=+64.782490307" watchObservedRunningTime="2026-07-16 17:35:34.359582644 +0000 UTC m=+64.783312705"
Jul 16 17:35:34.381400 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.381357    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4" podStartSLOduration=17.381342083 podStartE2EDuration="17.381342083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:34.380226218 +0000 UTC m=+64.803956278" watchObservedRunningTime="2026-07-16 17:35:34.381342083 +0000 UTC m=+64.805072142"
Jul 16 17:35:34.697728 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.697636    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:34.700270 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:34.700245    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:35.342499 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.342470    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:35.343681 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.343660    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-b8db8bc8d-dnsc4"
Jul 16 17:35:35.737880 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.737794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:35.738028 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.737923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:35.738028 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.737952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:35.740726 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.740704    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 17:35:35.740837 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.740748    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 17:35:35.740837 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.740778    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 17:35:35.751261 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.751237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/854d98e3-464a-4550-b809-e0964d0c8175-original-pull-secret\") pod \"global-pull-secret-syncer-4z97z\" (UID: \"854d98e3-464a-4550-b809-e0964d0c8175\") " pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:35.751360 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.751247    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f861f763-4c0a-4bfa-9880-aabe7948600b-metrics-certs\") pod \"network-metrics-daemon-48hrx\" (UID: \"f861f763-4c0a-4bfa-9880-aabe7948600b\") " pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:35.751660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.751645    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 17:35:35.761900 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.761879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhc76\" (UniqueName: \"kubernetes.io/projected/bb52c6e9-2de3-410b-ade0-87c4631fa276-kube-api-access-nhc76\") pod \"network-check-target-fqbvg\" (UID: \"bb52c6e9-2de3-410b-ade0-87c4631fa276\") " pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:35.799345 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.799321    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5khsp\""
Jul 16 17:35:35.807222 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.807200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:35.901660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.901636    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-vxx5g\""
Jul 16 17:35:35.904789 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.904763    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4z97z"
Jul 16 17:35:35.908510 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.908264    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48hrx"
Jul 16 17:35:35.937979 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:35.937941    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fqbvg"]
Jul 16 17:35:35.941179 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:35.941152    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb52c6e9_2de3_410b_ade0_87c4631fa276.slice/crio-9e42794f5dfb6a07672bf85fb6e561d2ba5e935f80a0f568f33826b69d96fc27 WatchSource:0}: Error finding container 9e42794f5dfb6a07672bf85fb6e561d2ba5e935f80a0f568f33826b69d96fc27: Status 404 returned error can't find the container with id 9e42794f5dfb6a07672bf85fb6e561d2ba5e935f80a0f568f33826b69d96fc27
Jul 16 17:35:36.040268 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:36.040168    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4z97z"]
Jul 16 17:35:36.042626 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:36.042599    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod854d98e3_464a_4550_b809_e0964d0c8175.slice/crio-31264ffa7a35c6c487dd1bf38bea422328057e304581a201297b5761d1d01487 WatchSource:0}: Error finding container 31264ffa7a35c6c487dd1bf38bea422328057e304581a201297b5761d1d01487: Status 404 returned error can't find the container with id 31264ffa7a35c6c487dd1bf38bea422328057e304581a201297b5761d1d01487
Jul 16 17:35:36.058346 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:36.058320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48hrx"]
Jul 16 17:35:36.061741 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:36.061721    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf861f763_4c0a_4bfa_9880_aabe7948600b.slice/crio-4ac650afb8d62c2bf30d5be93d8359c973388beecece66d428900167cb59232b WatchSource:0}: Error finding container 4ac650afb8d62c2bf30d5be93d8359c973388beecece66d428900167cb59232b: Status 404 returned error can't find the container with id 4ac650afb8d62c2bf30d5be93d8359c973388beecece66d428900167cb59232b
Jul 16 17:35:36.346668 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:36.346632    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48hrx" event={"ID":"f861f763-4c0a-4bfa-9880-aabe7948600b","Type":"ContainerStarted","Data":"4ac650afb8d62c2bf30d5be93d8359c973388beecece66d428900167cb59232b"}
Jul 16 17:35:36.347690 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:36.347668    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fqbvg" event={"ID":"bb52c6e9-2de3-410b-ade0-87c4631fa276","Type":"ContainerStarted","Data":"9e42794f5dfb6a07672bf85fb6e561d2ba5e935f80a0f568f33826b69d96fc27"}
Jul 16 17:35:36.348671 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:36.348647    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4z97z" event={"ID":"854d98e3-464a-4550-b809-e0964d0c8175","Type":"ContainerStarted","Data":"31264ffa7a35c6c487dd1bf38bea422328057e304581a201297b5761d1d01487"}
Jul 16 17:35:37.654473 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.654413    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:37.654927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.654505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:37.655428 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.655401    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:37.657254 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.657233    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-ts6vm\" (UID: \"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed\") " pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:37.755564 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.755531    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dpvg9\""
Jul 16 17:35:37.763506 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.763482    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-ts6vm"
Jul 16 17:35:37.856507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.856345    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:37.859985 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.859877    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7956263e-465c-4266-b46d-e271490872ed-metrics-tls\") pod \"dns-default-z4bnd\" (UID: \"7956263e-465c-4266-b46d-e271490872ed\") " pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:37.924408 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:37.924278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-ts6vm"]
Jul 16 17:35:37.930814 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:37.930789    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6c7f738_ca53_4aa2_a2bf_5ff5e6db20ed.slice/crio-d78f06c9c97828b771a3f2eacd3844b83fe4bb464eb171ba656252795ad63052 WatchSource:0}: Error finding container d78f06c9c97828b771a3f2eacd3844b83fe4bb464eb171ba656252795ad63052: Status 404 returned error can't find the container with id d78f06c9c97828b771a3f2eacd3844b83fe4bb464eb171ba656252795ad63052
Jul 16 17:35:38.096115 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.096087    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jn8p6\""
Jul 16 17:35:38.104181 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.104147    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:38.263618 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.263579    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z4bnd"]
Jul 16 17:35:38.365387 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.364972    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48hrx" event={"ID":"f861f763-4c0a-4bfa-9880-aabe7948600b","Type":"ContainerStarted","Data":"bb1fdde16f3d3d04f108ac3d6cebe5c374c61649a7a333bed1c4399761578ef7"}
Jul 16 17:35:38.365592 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.365467    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48hrx" event={"ID":"f861f763-4c0a-4bfa-9880-aabe7948600b","Type":"ContainerStarted","Data":"203e8dee0c139f921d7869875ce113c1a025da4c482a82e04b900ea8074329b9"}
Jul 16 17:35:38.366631 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.366605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts6vm" event={"ID":"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed","Type":"ContainerStarted","Data":"d78f06c9c97828b771a3f2eacd3844b83fe4bb464eb171ba656252795ad63052"}
Jul 16 17:35:38.384770 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:38.384706    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-48hrx" podStartSLOduration=67.19401695 podStartE2EDuration="1m8.384686412s" totalImagesPullingTime="1.190669462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:38.383942812 +0000 UTC m=+68.807672873" watchObservedRunningTime="2026-07-16 17:35:38.384686412 +0000 UTC m=+68.808416473"
Jul 16 17:35:39.207906 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:39.207875    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7956263e_465c_4266_b46d_e271490872ed.slice/crio-8c06bb2b1eb0a2ea6e25a5f378b6f378e0266dce5c8a35be93cebf5775223ba6 WatchSource:0}: Error finding container 8c06bb2b1eb0a2ea6e25a5f378b6f378e0266dce5c8a35be93cebf5775223ba6: Status 404 returned error can't find the container with id 8c06bb2b1eb0a2ea6e25a5f378b6f378e0266dce5c8a35be93cebf5775223ba6
Jul 16 17:35:39.372318 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:39.372274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bnd" event={"ID":"7956263e-465c-4266-b46d-e271490872ed","Type":"ContainerStarted","Data":"8c06bb2b1eb0a2ea6e25a5f378b6f378e0266dce5c8a35be93cebf5775223ba6"}
Jul 16 17:35:41.381730 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:41.381656    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fqbvg" event={"ID":"bb52c6e9-2de3-410b-ade0-87c4631fa276","Type":"ContainerStarted","Data":"11b068dbef37ccbf3eab8e9941ec0691c8ea16c0ac3eff2f82fc53de71d8635b"}
Jul 16 17:35:41.382194 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:41.381790    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:35:41.384317 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:41.384290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4z97z" event={"ID":"854d98e3-464a-4550-b809-e0964d0c8175","Type":"ContainerStarted","Data":"aa094d44b8803b9a0d16d897a8624b8f74e40fbc3840b5b8cb9e2a8a29ec4346"}
Jul 16 17:35:41.409051 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:41.408998    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fqbvg" podStartSLOduration=66.670911811 podStartE2EDuration="1m11.408979661s" totalImagesPullingTime="4.73806785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:41.408429458 +0000 UTC m=+71.832159517" watchObservedRunningTime="2026-07-16 17:35:41.408979661 +0000 UTC m=+71.832709721"
Jul 16 17:35:41.430727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:41.430679    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4z97z" podStartSLOduration=66.792471163 podStartE2EDuration="1m11.430665352s" totalImagesPullingTime="4.638194189s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:34:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:41.430273181 +0000 UTC m=+71.854003240" watchObservedRunningTime="2026-07-16 17:35:41.430665352 +0000 UTC m=+71.854395411"
Jul 16 17:35:43.394364 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:43.394330    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bnd" event={"ID":"7956263e-465c-4266-b46d-e271490872ed","Type":"ContainerStarted","Data":"6065109878a39a2c550be7b8cdad4b535207cb650b5a51cd436c6cb1a36376e0"}
Jul 16 17:35:46.410113 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:46.410070    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts6vm" event={"ID":"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed","Type":"ContainerStarted","Data":"dd8f0d5f620a9af02627732b8fab850bf57c6b25f8d9ff07d3a87b4eb7dfc12d"}
Jul 16 17:35:46.412088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:46.412052    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z4bnd" event={"ID":"7956263e-465c-4266-b46d-e271490872ed","Type":"ContainerStarted","Data":"862160ed6eecfaa36f7a4a95c8a45bdab74d181faaf58e5031da3bd331d3a99b"}
Jul 16 17:35:46.412316 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:46.412270    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:46.431278 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:46.431233    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-z4bnd" podStartSLOduration=38.122352401 podStartE2EDuration="41.43121581s" totalImagesPullingTime="3.308863409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:46.429866866 +0000 UTC m=+76.853596928" watchObservedRunningTime="2026-07-16 17:35:46.43121581 +0000 UTC m=+76.854945871"
Jul 16 17:35:49.425368 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:49.425323    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts6vm" event={"ID":"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed","Type":"ContainerStarted","Data":"3e6f693f80ca23304c6550eee391e3bf074b0b1f41af9740782717134abb6aab"}
Jul 16 17:35:51.546714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.546679    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"]
Jul 16 17:35:51.561542 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.561508    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:35:51.561690 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.561650    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.568696 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.568567    2576 patch_prober.go:28] interesting pod/image-registry-548664f95f-jxnsj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 17:35:51.568696 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.568633    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:35:51.573595 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.573553    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 17:35:51.574084 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.574062    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 17:35:51.574198 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.574163    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 17:35:51.574198 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.574163    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 17:35:51.574603 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.574359    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 17:35:51.579860 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.579843    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 17:35:51.580501 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.580484    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 17:35:51.601540 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.601506    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"]
Jul 16 17:35:51.666210 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666126    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.666210 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666195    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.666483 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666228    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e6f00e-2c91-404c-a138-426aa589e4f2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.666483 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.666483 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666284    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8lvk\" (UniqueName: \"kubernetes.io/projected/05e6f00e-2c91-404c-a138-426aa589e4f2-kube-api-access-v8lvk\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.666483 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.666318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.766874 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.766874 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e6f00e-2c91-404c-a138-426aa589e4f2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.767112 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.767112 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8lvk\" (UniqueName: \"kubernetes.io/projected/05e6f00e-2c91-404c-a138-426aa589e4f2-kube-api-access-v8lvk\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.767112 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.767112 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.766996    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.767703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.767673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e6f00e-2c91-404c-a138-426aa589e4f2-additional-service-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.770423 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.769977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.770423 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.770218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.770423 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.770382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.770688 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.770604    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/05e6f00e-2c91-404c-a138-426aa589e4f2-ca\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.794128 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.794099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8lvk\" (UniqueName: \"kubernetes.io/projected/05e6f00e-2c91-404c-a138-426aa589e4f2-kube-api-access-v8lvk\") pod \"cluster-proxy-proxy-agent-779b9786d9-dm7n4\" (UID: \"05e6f00e-2c91-404c-a138-426aa589e4f2\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:51.884127 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:51.884088    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"
Jul 16 17:35:52.512181 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:52.512153    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4"]
Jul 16 17:35:52.515409 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:35:52.515386    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05e6f00e_2c91_404c_a138_426aa589e4f2.slice/crio-920877ee6194aec6b0ab08b6d8392b3dbb6b3c47fa71781d84f6893e9515eda5 WatchSource:0}: Error finding container 920877ee6194aec6b0ab08b6d8392b3dbb6b3c47fa71781d84f6893e9515eda5: Status 404 returned error can't find the container with id 920877ee6194aec6b0ab08b6d8392b3dbb6b3c47fa71781d84f6893e9515eda5
Jul 16 17:35:53.418521 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:53.418489    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z4bnd"
Jul 16 17:35:53.443113 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:53.443075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-ts6vm" event={"ID":"b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed","Type":"ContainerStarted","Data":"88e2140bde6a585a65ff0050971d467cc47be49df89919b0277edf4cdc921742"}
Jul 16 17:35:53.444635 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:53.444592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4" event={"ID":"05e6f00e-2c91-404c-a138-426aa589e4f2","Type":"ContainerStarted","Data":"920877ee6194aec6b0ab08b6d8392b3dbb6b3c47fa71781d84f6893e9515eda5"}
Jul 16 17:35:56.457495 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:56.457457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4" event={"ID":"05e6f00e-2c91-404c-a138-426aa589e4f2","Type":"ContainerStarted","Data":"b8aa7c28d8eaa807f794d73807747aadf624421fa04d590d3cc50e1da275ddf0"}
Jul 16 17:35:56.457870 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:56.457510    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4" event={"ID":"05e6f00e-2c91-404c-a138-426aa589e4f2","Type":"ContainerStarted","Data":"60432019a83546be05b7b7b14a4aa9ba5068466083f484358162af3e074b80d7"}
Jul 16 17:35:56.457870 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:56.457529    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4" event={"ID":"05e6f00e-2c91-404c-a138-426aa589e4f2","Type":"ContainerStarted","Data":"aaa7888f68daa03830b30c5d25a5f941c6bc4ac5c1aac70442281585b4e4c2a4"}
Jul 16 17:35:56.478040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:56.477990    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-ts6vm" podStartSLOduration=21.095441977 podStartE2EDuration="35.47797619s" totalImagesPullingTime="14.382534213s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 17:35:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:53.466749036 +0000 UTC m=+83.890479099" watchObservedRunningTime="2026-07-16 17:35:56.47797619 +0000 UTC m=+86.901706251"
Jul 16 17:35:56.478263 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:35:56.478245    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-779b9786d9-dm7n4" podStartSLOduration=1.956271028 podStartE2EDuration="5.478240874s" totalImagesPullingTime="3.521969846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:35:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:35:56.47739934 +0000 UTC m=+86.901129400" watchObservedRunningTime="2026-07-16 17:35:56.478240874 +0000 UTC m=+86.901970934"
Jul 16 17:36:01.565886 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:01.565858    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:36:12.392590 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:12.392560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fqbvg"
Jul 16 17:36:16.588344 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:16.588285    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerName="registry" containerID="cri-o://5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886" gracePeriod=30
Jul 16 17:36:16.876845 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:16.876818    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:36:17.056008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.055973    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056006    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056257 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056052    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056257 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056072    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056257 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056100    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69w94\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056257 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056167    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056481 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056309    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.056481 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056361    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") pod \"05c6e21e-4d56-437d-a651-c35b2be32e4e\" (UID: \"05c6e21e-4d56-437d-a651-c35b2be32e4e\") "
Jul 16 17:36:17.057211 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.056594    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:17.057345 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.057294    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:36:17.059374 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.059316    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:17.061490 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.059657    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:17.061490 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.059709    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94" (OuterVolumeSpecName: "kube-api-access-69w94") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "kube-api-access-69w94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:17.061490 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.059872    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:36:17.061490 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.059887    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:36:17.066737 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.066711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "05c6e21e-4d56-437d-a651-c35b2be32e4e" (UID: "05c6e21e-4d56-437d-a651-c35b2be32e4e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:36:17.085635 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.085594    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cc6sv"]
Jul 16 17:36:17.086056 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.086043    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerName="registry"
Jul 16 17:36:17.086120 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.086057    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerName="registry"
Jul 16 17:36:17.086120 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.086093    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerName="registry"
Jul 16 17:36:17.121905 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.121778    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cc6sv"]
Jul 16 17:36:17.122068 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.121901    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.124493 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124446    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 17:36:17.124493 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124458    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 17:36:17.124691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124584    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 17:36:17.124691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124616    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 17:36:17.124691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124643    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 17:36:17.124691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.124670    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-8b855\""
Jul 16 17:36:17.157691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157657    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05c6e21e-4d56-437d-a651-c35b2be32e4e-ca-trust-extracted\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157685    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-trusted-ca\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157691 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157696    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157704    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05c6e21e-4d56-437d-a651-c35b2be32e4e-registry-certificates\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157712    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-bound-sa-token\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157720    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-installation-pull-secrets\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157728    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05c6e21e-4d56-437d-a651-c35b2be32e4e-image-registry-private-configuration\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.157917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.157738    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69w94\" (UniqueName: \"kubernetes.io/projected/05c6e21e-4d56-437d-a651-c35b2be32e4e-kube-api-access-69w94\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:36:17.258608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.258557    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.258608 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.258607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.258873 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.258650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/516ab859-9ccd-4464-bdf8-9b367616d872-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.258873 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.258740    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ljn2\" (UniqueName: \"kubernetes.io/projected/516ab859-9ccd-4464-bdf8-9b367616d872-kube-api-access-6ljn2\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.359128 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.359096    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.359128 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.359127    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.359323 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.359145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/516ab859-9ccd-4464-bdf8-9b367616d872-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.359323 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.359178    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ljn2\" (UniqueName: \"kubernetes.io/projected/516ab859-9ccd-4464-bdf8-9b367616d872-kube-api-access-6ljn2\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.359966 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.359907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/516ab859-9ccd-4464-bdf8-9b367616d872-metrics-client-ca\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.361618 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.361595    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.361618 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.361609    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/516ab859-9ccd-4464-bdf8-9b367616d872-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.369170 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.369153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ljn2\" (UniqueName: \"kubernetes.io/projected/516ab859-9ccd-4464-bdf8-9b367616d872-kube-api-access-6ljn2\") pod \"prometheus-operator-64785b6675-cc6sv\" (UID: \"516ab859-9ccd-4464-bdf8-9b367616d872\") " pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.431016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.430965    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv"
Jul 16 17:36:17.522536 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.522505    2576 generic.go:356] "Generic (PLEG): container finished" podID="05c6e21e-4d56-437d-a651-c35b2be32e4e" containerID="5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886" exitCode=0
Jul 16 17:36:17.522659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.522550    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" event={"ID":"05c6e21e-4d56-437d-a651-c35b2be32e4e","Type":"ContainerDied","Data":"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"}
Jul 16 17:36:17.522659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.522570    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-548664f95f-jxnsj"
Jul 16 17:36:17.522659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.522607    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-548664f95f-jxnsj" event={"ID":"05c6e21e-4d56-437d-a651-c35b2be32e4e","Type":"ContainerDied","Data":"969af2cd5fc0ceb74774117d123d38ea0126e4e55ece50d1d48e3528ceeae6b0"}
Jul 16 17:36:17.522659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.522629    2576 scope.go:117] "RemoveContainer" containerID="5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"
Jul 16 17:36:17.531929 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.531909    2576 scope.go:117] "RemoveContainer" containerID="5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"
Jul 16 17:36:17.532193 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:36:17.532176    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886\": container with ID starting with 5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886 not found: ID does not exist" containerID="5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"
Jul 16 17:36:17.532245 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.532201    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886"} err="failed to get container status \"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886\": rpc error: code = NotFound desc = could not find container \"5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886\": container with ID starting with 5457f401d358be9bd3733045991d0660568e4a308e30c8426edda5e58b177886 not found: ID does not exist"
Jul 16 17:36:17.544519 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.544498    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:36:17.549273 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.549250    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-548664f95f-jxnsj"]
Jul 16 17:36:17.588959 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:17.588924    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-cc6sv"]
Jul 16 17:36:17.593544 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:36:17.593522    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod516ab859_9ccd_4464_bdf8_9b367616d872.slice/crio-45b4c7792f9183efa98f0d8f849327f390898107fc394757fb9c8db495e82c73 WatchSource:0}: Error finding container 45b4c7792f9183efa98f0d8f849327f390898107fc394757fb9c8db495e82c73: Status 404 returned error can't find the container with id 45b4c7792f9183efa98f0d8f849327f390898107fc394757fb9c8db495e82c73
Jul 16 17:36:18.090510 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:18.090474    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05c6e21e-4d56-437d-a651-c35b2be32e4e" path="/var/lib/kubelet/pods/05c6e21e-4d56-437d-a651-c35b2be32e4e/volumes"
Jul 16 17:36:18.527047 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:18.526966    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv" event={"ID":"516ab859-9ccd-4464-bdf8-9b367616d872","Type":"ContainerStarted","Data":"45b4c7792f9183efa98f0d8f849327f390898107fc394757fb9c8db495e82c73"}
Jul 16 17:36:20.536660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:20.536618    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv" event={"ID":"516ab859-9ccd-4464-bdf8-9b367616d872","Type":"ContainerStarted","Data":"d9b74b2f9aff13c5f10619f4584543018a03ef8b2fd418f044094829acb7d776"}
Jul 16 17:36:20.537008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:20.536678    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv" event={"ID":"516ab859-9ccd-4464-bdf8-9b367616d872","Type":"ContainerStarted","Data":"3ced77877794c9c5f710651fc4e1b3e339434c8b15a142ca58a5a1bb4236789e"}
Jul 16 17:36:20.557539 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:20.557495    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-cc6sv" podStartSLOduration=1.526006782 podStartE2EDuration="3.557481832s" totalImagesPullingTime="2.03147505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:20.556817845 +0000 UTC m=+110.980547907" watchObservedRunningTime="2026-07-16 17:36:20.557481832 +0000 UTC m=+110.981211892"
Jul 16 17:36:22.536669 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.536635    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-t8mr2"]
Jul 16 17:36:22.617602 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.617567    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.619985 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.619959    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 17:36:22.620112 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.619963    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-frvd5\""
Jul 16 17:36:22.620165 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.620139    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 17:36:22.620301 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.620280    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 17:36:22.801042 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.800967    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-metrics-client-ca\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801042 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801010    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e883712-9c0c-452e-a959-380785443207-node-exporter-textfile\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801042 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801028    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801042 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-node-exporter-wtmp\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801330 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801330 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm6fj\" (UniqueName: \"kubernetes.io/projected/6e883712-9c0c-452e-a959-380785443207-kube-api-access-pm6fj\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801330 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801181    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-sys\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801330 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801204    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.801330 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.801239    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-root\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902603 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902568    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e883712-9c0c-452e-a959-380785443207-node-exporter-textfile\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902603 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902830 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-node-exporter-wtmp\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902830 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902830 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm6fj\" (UniqueName: \"kubernetes.io/projected/6e883712-9c0c-452e-a959-380785443207-kube-api-access-pm6fj\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.902830 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-sys\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902852    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-root\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902915    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-node-exporter-wtmp\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6e883712-9c0c-452e-a959-380785443207-node-exporter-textfile\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.902959    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-sys\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903016 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:36:22.903010    2576 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 17:36:22.903284 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.903060    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6e883712-9c0c-452e-a959-380785443207-root\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903284 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:36:22.903087    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls podName:6e883712-9c0c-452e-a959-380785443207 nodeName:}" failed. No retries permitted until 2026-07-16 17:36:23.403059446 +0000 UTC m=+113.826789498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls") pod "node-exporter-t8mr2" (UID: "6e883712-9c0c-452e-a959-380785443207") : secret "node-exporter-tls" not found
Jul 16 17:36:22.903284 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.903146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-metrics-client-ca\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903399 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.903339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-node-exporter-accelerators-collector-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.903880 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.903858    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6e883712-9c0c-452e-a959-380785443207-metrics-client-ca\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.905152 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.905133    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:22.915693 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:22.915673    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm6fj\" (UniqueName: \"kubernetes.io/projected/6e883712-9c0c-452e-a959-380785443207-kube-api-access-pm6fj\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:23.406991 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:23.406956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:23.409372 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:23.409341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6e883712-9c0c-452e-a959-380785443207-node-exporter-tls\") pod \"node-exporter-t8mr2\" (UID: \"6e883712-9c0c-452e-a959-380785443207\") " pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:23.526512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:23.526476    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-t8mr2"
Jul 16 17:36:23.542027 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:36:23.541990    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e883712_9c0c_452e_a959_380785443207.slice/crio-849c5b4744447e7d07f2ef4cc538bf4ccf47d0cd21815e6eea0b1a436dbd0382 WatchSource:0}: Error finding container 849c5b4744447e7d07f2ef4cc538bf4ccf47d0cd21815e6eea0b1a436dbd0382: Status 404 returned error can't find the container with id 849c5b4744447e7d07f2ef4cc538bf4ccf47d0cd21815e6eea0b1a436dbd0382
Jul 16 17:36:23.551709 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:23.551674    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8mr2" event={"ID":"6e883712-9c0c-452e-a959-380785443207","Type":"ContainerStarted","Data":"849c5b4744447e7d07f2ef4cc538bf4ccf47d0cd21815e6eea0b1a436dbd0382"}
Jul 16 17:36:24.557192 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:24.557159    2576 generic.go:356] "Generic (PLEG): container finished" podID="6e883712-9c0c-452e-a959-380785443207" containerID="54394e471511bee8894b4b6484a4dd389ea0d084616c43dc99e6db2951f16ce3" exitCode=0
Jul 16 17:36:24.557540 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:24.557247    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8mr2" event={"ID":"6e883712-9c0c-452e-a959-380785443207","Type":"ContainerDied","Data":"54394e471511bee8894b4b6484a4dd389ea0d084616c43dc99e6db2951f16ce3"}
Jul 16 17:36:25.563556 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:25.563508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8mr2" event={"ID":"6e883712-9c0c-452e-a959-380785443207","Type":"ContainerStarted","Data":"f71641adb03ca9a3c5bfe1ed0a632e0cace6e0b504ef4745d9d2a931be4a09d8"}
Jul 16 17:36:25.563959 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:25.563574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-t8mr2" event={"ID":"6e883712-9c0c-452e-a959-380785443207","Type":"ContainerStarted","Data":"2ba31b8777eb954b0ef2bf22c706d27c8cda76f7106d861aff208d82b857db92"}
Jul 16 17:36:25.585395 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:25.585348    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-t8mr2" podStartSLOduration=2.639845377 podStartE2EDuration="3.585332484s" totalImagesPullingTime="873.490154ms" totalInitContainerRuntime="71.996953ms" isStatefulPod=true podCreationTimestamp="2026-07-16 17:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:25.583582227 +0000 UTC m=+116.007312287" watchObservedRunningTime="2026-07-16 17:36:25.585332484 +0000 UTC m=+116.009062575"
Jul 16 17:36:26.807941 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.807910    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-55d48d7796-bvgzx"]
Jul 16 17:36:26.809920 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.809904    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.812322 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.812283    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 17:36:26.812322 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.812285    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 17:36:26.813396 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.813374    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-k425b\""
Jul 16 17:36:26.813525 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.813416    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-bbohu4lrumll8\""
Jul 16 17:36:26.813525 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.813483    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 17:36:26.813525 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.813507    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 17:36:26.823149 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.823122    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55d48d7796-bvgzx"]
Jul 16 17:36:26.933573 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933542    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-metrics-server-audit-profiles\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933573 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933575    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw8cn\" (UniqueName: \"kubernetes.io/projected/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-kube-api-access-rw8cn\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-client-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933670    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-tls\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933700    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-client-certs\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:26.933803 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:26.933744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-audit-log\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.034878 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.034840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-metrics-server-audit-profiles\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.034878 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.034879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw8cn\" (UniqueName: \"kubernetes.io/projected/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-kube-api-access-rw8cn\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035096 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.034918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-client-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035096 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.034941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035096 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.034971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-tls\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035096 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.035000    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-client-certs\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035096 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.035030    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-audit-log\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035495 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.035471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-audit-log\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.035944 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.035920    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.036041 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.035972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-metrics-server-audit-profiles\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.037622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.037601    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-tls\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.037729 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.037713    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-secret-metrics-server-client-certs\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.037834 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.037817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-client-ca-bundle\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.048052 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.048033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw8cn\" (UniqueName: \"kubernetes.io/projected/34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc-kube-api-access-rw8cn\") pod \"metrics-server-55d48d7796-bvgzx\" (UID: \"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc\") " pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.118891 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.118855    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:27.261000 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.260973    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-55d48d7796-bvgzx"]
Jul 16 17:36:27.263663 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:36:27.263636    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34dd1bf1_45c9_4a2b_a74d_7ea2b49243bc.slice/crio-70d1e17a4ae6cc3a31bf055274730df963f042014867d62837033f1a29f0b8f6 WatchSource:0}: Error finding container 70d1e17a4ae6cc3a31bf055274730df963f042014867d62837033f1a29f0b8f6: Status 404 returned error can't find the container with id 70d1e17a4ae6cc3a31bf055274730df963f042014867d62837033f1a29f0b8f6
Jul 16 17:36:27.263787 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.263760    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"]
Jul 16 17:36:27.266306 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.266288    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.268878 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.268857    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 17:36:27.269161 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.269132    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-7mr22\""
Jul 16 17:36:27.278322 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.278294    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"]
Jul 16 17:36:27.440452 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.440343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.440452 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.440377    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ntt7\" (UniqueName: \"kubernetes.io/projected/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-kube-api-access-5ntt7\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.541040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.541009    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.541196 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.541047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ntt7\" (UniqueName: \"kubernetes.io/projected/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-kube-api-access-5ntt7\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.543542 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.543511    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-monitoring-plugin-cert\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.550770 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.550742    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ntt7\" (UniqueName: \"kubernetes.io/projected/b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8-kube-api-access-5ntt7\") pod \"monitoring-plugin-5f68df7f58-tt5b7\" (UID: \"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8\") " pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.571468 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.571417    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx" event={"ID":"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc","Type":"ContainerStarted","Data":"70d1e17a4ae6cc3a31bf055274730df963f042014867d62837033f1a29f0b8f6"}
Jul 16 17:36:27.575620 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.575606    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:27.712412 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:27.712333    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"]
Jul 16 17:36:27.715609 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:36:27.715579    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a3ce9b_4e6b_4393_9198_ba4ee1c66ff8.slice/crio-4b9e5315ec1bbea85c69c49facbcbe94cbd30109c75fd7715ec14de9efd320f4 WatchSource:0}: Error finding container 4b9e5315ec1bbea85c69c49facbcbe94cbd30109c75fd7715ec14de9efd320f4: Status 404 returned error can't find the container with id 4b9e5315ec1bbea85c69c49facbcbe94cbd30109c75fd7715ec14de9efd320f4
Jul 16 17:36:28.576928 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:28.576683    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx" event={"ID":"34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc","Type":"ContainerStarted","Data":"abc19462f5b07997c7d5215484f8609f0ab4d9acca0695b114b50509d21d9da9"}
Jul 16 17:36:28.578522 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:28.578484    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7" event={"ID":"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8","Type":"ContainerStarted","Data":"4b9e5315ec1bbea85c69c49facbcbe94cbd30109c75fd7715ec14de9efd320f4"}
Jul 16 17:36:28.605012 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:28.604953    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx" podStartSLOduration=1.4344663610000001 podStartE2EDuration="2.604936606s" totalImagesPullingTime="1.170470245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:28.604670739 +0000 UTC m=+119.028400800" watchObservedRunningTime="2026-07-16 17:36:28.604936606 +0000 UTC m=+119.028666666"
Jul 16 17:36:34.600331 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:34.600292    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7" event={"ID":"b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8","Type":"ContainerStarted","Data":"e708dd912d3667ca309139fd369b9a67e5061639e8d8e6b14102eae270e83e33"}
Jul 16 17:36:34.600781 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:34.600460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:34.605250 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:34.605228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7"
Jul 16 17:36:34.621570 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:34.621525    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5f68df7f58-tt5b7" podStartSLOduration=1.103346827 podStartE2EDuration="7.621511472s" totalImagesPullingTime="6.518164645s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:36:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:36:34.620971019 +0000 UTC m=+125.044701079" watchObservedRunningTime="2026-07-16 17:36:34.621511472 +0000 UTC m=+125.045241532"
Jul 16 17:36:46.642387 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:46.642355    2576 generic.go:356] "Generic (PLEG): container finished" podID="3dbcd895-ffcb-4eb3-be2b-871643f6110b" containerID="e616064095f19b0b31e18adcb19233d02e5cfde2c839e3c9dfff5b0082bdf8cb" exitCode=0
Jul 16 17:36:46.642789 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:46.642412    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7mght" event={"ID":"3dbcd895-ffcb-4eb3-be2b-871643f6110b","Type":"ContainerDied","Data":"e616064095f19b0b31e18adcb19233d02e5cfde2c839e3c9dfff5b0082bdf8cb"}
Jul 16 17:36:46.642789 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:46.642765    2576 scope.go:117] "RemoveContainer" containerID="e616064095f19b0b31e18adcb19233d02e5cfde2c839e3c9dfff5b0082bdf8cb"
Jul 16 17:36:47.119892 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:47.119847    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:47.120055 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:47.119918    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:36:47.649812 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:36:47.649784    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7mght" event={"ID":"3dbcd895-ffcb-4eb3-be2b-871643f6110b","Type":"ContainerStarted","Data":"132a5b2eb52ac8bd2548c0a4757c9f5899a00678855da47569e8c44561c83043"}
Jul 16 17:37:07.124686 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:37:07.124658    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:37:07.128424 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:37:07.128402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-55d48d7796-bvgzx"
Jul 16 17:39:29.922703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:39:29.922676    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:39:29.924813 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:39:29.924790    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:39:29.929244 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:39:29.929222    2576 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 17:41:05.541118 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.541076    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"]
Jul 16 17:41:05.543993 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.543974    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.547313 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.547155    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 17:41:05.548841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.547499    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 16 17:41:05.548841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.548110    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 17:41:05.548841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.548271    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 17:41:05.548841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.548400    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-xxpq2\""
Jul 16 17:41:05.548841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.548677    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 17:41:05.555269 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.555244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"]
Jul 16 17:41:05.684819 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.684780    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvmkm\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-kube-api-access-xvmkm\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.684819 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.684825    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/46d9480e-cd7a-40fe-9655-8f5f62219bd1-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.685040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.684851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-certificates\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.785566 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.785523    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvmkm\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-kube-api-access-xvmkm\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.785753 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.785578    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/46d9480e-cd7a-40fe-9655-8f5f62219bd1-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.785753 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.785612    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-certificates\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.786026 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.786003    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/46d9480e-cd7a-40fe-9655-8f5f62219bd1-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.788273 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.788250    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-certificates\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.796067 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.795997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvmkm\" (UniqueName: \"kubernetes.io/projected/46d9480e-cd7a-40fe-9655-8f5f62219bd1-kube-api-access-xvmkm\") pod \"keda-metrics-apiserver-55557df8c-m6vs7\" (UID: \"46d9480e-cd7a-40fe-9655-8f5f62219bd1\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.846834 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.846801    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-88sfp"]
Jul 16 17:41:05.849981 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.849964    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:05.852862 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.852839    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 16 17:41:05.856842 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.856823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:05.878286 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.878259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-88sfp"]
Jul 16 17:41:05.987153 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.987123    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-certificates\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:05.987324 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:05.987241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htjqx\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-kube-api-access-htjqx\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.007970 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.007937    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"]
Jul 16 17:41:06.012228 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:41:06.012198    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46d9480e_cd7a_40fe_9655_8f5f62219bd1.slice/crio-1e79fcb958be05789fb3137354ae38df7f4f4f8f1b76e0f1a3a5163e9ea5532d WatchSource:0}: Error finding container 1e79fcb958be05789fb3137354ae38df7f4f4f8f1b76e0f1a3a5163e9ea5532d: Status 404 returned error can't find the container with id 1e79fcb958be05789fb3137354ae38df7f4f4f8f1b76e0f1a3a5163e9ea5532d
Jul 16 17:41:06.016281 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.016261    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:41:06.087981 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.087944    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-certificates\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.088158 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.088016    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htjqx\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-kube-api-access-htjqx\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.090774 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.090749    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-certificates\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.096481 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.096457    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htjqx\" (UniqueName: \"kubernetes.io/projected/bffaa295-df43-4ee5-80dd-bd7f02544625-kube-api-access-htjqx\") pod \"keda-admission-56f9798bb4-88sfp\" (UID: \"bffaa295-df43-4ee5-80dd-bd7f02544625\") " pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.160507 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.160466    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:06.317219 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.317188    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-88sfp"]
Jul 16 17:41:06.321231 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:41:06.321208    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbffaa295_df43_4ee5_80dd_bd7f02544625.slice/crio-2cffcee3e7a634ddd36ba6dd2509746190811dae15f6fe01a355268d1b4b0163 WatchSource:0}: Error finding container 2cffcee3e7a634ddd36ba6dd2509746190811dae15f6fe01a355268d1b4b0163: Status 404 returned error can't find the container with id 2cffcee3e7a634ddd36ba6dd2509746190811dae15f6fe01a355268d1b4b0163
Jul 16 17:41:06.553868 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.553779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-88sfp" event={"ID":"bffaa295-df43-4ee5-80dd-bd7f02544625","Type":"ContainerStarted","Data":"2cffcee3e7a634ddd36ba6dd2509746190811dae15f6fe01a355268d1b4b0163"}
Jul 16 17:41:06.554883 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:06.554860    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7" event={"ID":"46d9480e-cd7a-40fe-9655-8f5f62219bd1","Type":"ContainerStarted","Data":"1e79fcb958be05789fb3137354ae38df7f4f4f8f1b76e0f1a3a5163e9ea5532d"}
Jul 16 17:41:09.569229 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.569198    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-88sfp" event={"ID":"bffaa295-df43-4ee5-80dd-bd7f02544625","Type":"ContainerStarted","Data":"1f26fd5b4aa4c6ff7aecca66281e06eeb216d8054093f383de1ad4ee5129de24"}
Jul 16 17:41:09.569663 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.569312    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:41:09.570927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.570899    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7" event={"ID":"46d9480e-cd7a-40fe-9655-8f5f62219bd1","Type":"ContainerStarted","Data":"6178dac2e42bd54cde7db116a48dc70261ad6a9c751fe2703b94e31fb15f41a1"}
Jul 16 17:41:09.571055 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.570958    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:09.588891 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.588836    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-88sfp" podStartSLOduration=1.418639373 podStartE2EDuration="4.58881657s" totalImagesPullingTime="3.170177197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:09.588108007 +0000 UTC m=+400.011838068" watchObservedRunningTime="2026-07-16 17:41:09.58881657 +0000 UTC m=+400.012546632"
Jul 16 17:41:09.605273 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:09.605219    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7" podStartSLOduration=1.127628467 podStartE2EDuration="4.60519931s" totalImagesPullingTime="3.477570843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:41:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:41:09.604700187 +0000 UTC m=+400.028430249" watchObservedRunningTime="2026-07-16 17:41:09.60519931 +0000 UTC m=+400.028929371"
Jul 16 17:41:20.579705 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:20.579677    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-m6vs7"
Jul 16 17:41:30.577149 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:41:30.577118    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-88sfp"
Jul 16 17:42:13.671120 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.671045    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-scsz5"]
Jul 16 17:42:13.673167 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.673152    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.675843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.675818    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 17:42:13.675952 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.675829    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-2cm5w\""
Jul 16 17:42:13.676638 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.676617    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 17:42:13.676740 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.676617    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 17:42:13.683808 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.683789    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-scsz5"]
Jul 16 17:42:13.770832 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.770796    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8ggh\" (UniqueName: \"kubernetes.io/projected/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-kube-api-access-h8ggh\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.770832 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.770847    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-data\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.871512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.871482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8ggh\" (UniqueName: \"kubernetes.io/projected/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-kube-api-access-h8ggh\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.871698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.871522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-data\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.871867 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.871852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-data\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.880474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.880423    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8ggh\" (UniqueName: \"kubernetes.io/projected/bfc329ba-0dbe-4ad5-b2af-906d8d05aa53-kube-api-access-h8ggh\") pod \"seaweedfs-64568bcd49-scsz5\" (UID: \"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53\") " pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:13.982334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:13.982238    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:14.129936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:14.129906    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-scsz5"]
Jul 16 17:42:14.134072 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:42:14.134045    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc329ba_0dbe_4ad5_b2af_906d8d05aa53.slice/crio-f6606390bab52c7caa6529231ae7ffa222e731b656ba80a92e6eb5d8af446c1d WatchSource:0}: Error finding container f6606390bab52c7caa6529231ae7ffa222e731b656ba80a92e6eb5d8af446c1d: Status 404 returned error can't find the container with id f6606390bab52c7caa6529231ae7ffa222e731b656ba80a92e6eb5d8af446c1d
Jul 16 17:42:14.801722 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:14.801684    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-scsz5" event={"ID":"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53","Type":"ContainerStarted","Data":"f6606390bab52c7caa6529231ae7ffa222e731b656ba80a92e6eb5d8af446c1d"}
Jul 16 17:42:17.817238 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:17.816969    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-scsz5" event={"ID":"bfc329ba-0dbe-4ad5-b2af-906d8d05aa53","Type":"ContainerStarted","Data":"8b92b436fd8198d21bda550a31f50da718f5da02e135c45b2bfa7c0a53f4bdbb"}
Jul 16 17:42:17.817698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:17.817286    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:42:17.843308 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:17.843262    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-scsz5" podStartSLOduration=1.670428379 podStartE2EDuration="4.843247272s" totalImagesPullingTime="3.172818893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:42:17.841422685 +0000 UTC m=+468.265152744" watchObservedRunningTime="2026-07-16 17:42:17.843247272 +0000 UTC m=+468.266977331"
Jul 16 17:42:23.823137 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:42:23.823104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-scsz5"
Jul 16 17:43:32.128588 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.128508    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-fd65d7d6-7xg4w"]
Jul 16 17:43:32.131674 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.131658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.133986 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.133969    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 16 17:43:32.134072 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.133985    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-5llpw\""
Jul 16 17:43:32.139538 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.139516    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-84b54b7f97-knwdl"]
Jul 16 17:43:32.142321 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.142304    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.143026 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.143004    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-7xg4w"]
Jul 16 17:43:32.144816 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.144797    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 16 17:43:32.144918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.144869    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-xhgmz\""
Jul 16 17:43:32.159226 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.159201    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-knwdl"]
Jul 16 17:43:32.229759 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.229717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9xm2\" (UniqueName: \"kubernetes.io/projected/9fbdcd3a-982e-4856-b420-c720bd6cbcee-kube-api-access-p9xm2\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.229759 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.229760    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7xfq\" (UniqueName: \"kubernetes.io/projected/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-kube-api-access-m7xfq\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.229975 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.229844    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.229975 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.229877    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fbdcd3a-982e-4856-b420-c720bd6cbcee-cert\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.330593 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.330562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.330593 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.330601    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fbdcd3a-982e-4856-b420-c720bd6cbcee-cert\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.330861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.330643    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9xm2\" (UniqueName: \"kubernetes.io/projected/9fbdcd3a-982e-4856-b420-c720bd6cbcee-kube-api-access-p9xm2\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.330861 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.330659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7xfq\" (UniqueName: \"kubernetes.io/projected/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-kube-api-access-m7xfq\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.330861 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:43:32.330731    2576 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 16 17:43:32.330861 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:43:32.330815    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs podName:0a1c60c2-1bc1-4056-8a05-ad70b5fc7650 nodeName:}" failed. No retries permitted until 2026-07-16 17:43:32.830795761 +0000 UTC m=+543.254525802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs") pod "model-serving-api-fd65d7d6-7xg4w" (UID: "0a1c60c2-1bc1-4056-8a05-ad70b5fc7650") : secret "model-serving-api-tls" not found
Jul 16 17:43:32.333196 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.333175    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fbdcd3a-982e-4856-b420-c720bd6cbcee-cert\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.343130 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.343101    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9xm2\" (UniqueName: \"kubernetes.io/projected/9fbdcd3a-982e-4856-b420-c720bd6cbcee-kube-api-access-p9xm2\") pod \"odh-model-controller-84b54b7f97-knwdl\" (UID: \"9fbdcd3a-982e-4856-b420-c720bd6cbcee\") " pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.343246 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.343231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7xfq\" (UniqueName: \"kubernetes.io/projected/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-kube-api-access-m7xfq\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.452808 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.452719    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:32.584469 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.584417    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-84b54b7f97-knwdl"]
Jul 16 17:43:32.587523 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:43:32.587499    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbdcd3a_982e_4856_b420_c720bd6cbcee.slice/crio-9b3b376f80e310cf36340789d140bf0da212fdc668913752365b59b1d3dbe0e4 WatchSource:0}: Error finding container 9b3b376f80e310cf36340789d140bf0da212fdc668913752365b59b1d3dbe0e4: Status 404 returned error can't find the container with id 9b3b376f80e310cf36340789d140bf0da212fdc668913752365b59b1d3dbe0e4
Jul 16 17:43:32.835501 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.835467    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:32.837952 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:32.837934    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a1c60c2-1bc1-4056-8a05-ad70b5fc7650-tls-certs\") pod \"model-serving-api-fd65d7d6-7xg4w\" (UID: \"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650\") " pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:33.041277 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:33.041236    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:33.082770 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:33.082731    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-knwdl" event={"ID":"9fbdcd3a-982e-4856-b420-c720bd6cbcee","Type":"ContainerStarted","Data":"9b3b376f80e310cf36340789d140bf0da212fdc668913752365b59b1d3dbe0e4"}
Jul 16 17:43:33.176066 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:33.176032    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-fd65d7d6-7xg4w"]
Jul 16 17:43:33.178840 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:43:33.178814    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1c60c2_1bc1_4056_8a05_ad70b5fc7650.slice/crio-49225dda45daab8c48107ea116ff4777222e1cf7efed18e1dba4513ae37760ce WatchSource:0}: Error finding container 49225dda45daab8c48107ea116ff4777222e1cf7efed18e1dba4513ae37760ce: Status 404 returned error can't find the container with id 49225dda45daab8c48107ea116ff4777222e1cf7efed18e1dba4513ae37760ce
Jul 16 17:43:34.094588 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:34.094541    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-7xg4w" event={"ID":"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650","Type":"ContainerStarted","Data":"49225dda45daab8c48107ea116ff4777222e1cf7efed18e1dba4513ae37760ce"}
Jul 16 17:43:37.109628 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.109591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-fd65d7d6-7xg4w" event={"ID":"0a1c60c2-1bc1-4056-8a05-ad70b5fc7650","Type":"ContainerStarted","Data":"89e2572524eaa10d22ded6b3ed83aababfc4a22d4e0d91f5dbbc1cc8db5bbbaa"}
Jul 16 17:43:37.110124 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.109703    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:43:37.111450 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.111408    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-84b54b7f97-knwdl" event={"ID":"9fbdcd3a-982e-4856-b420-c720bd6cbcee","Type":"ContainerStarted","Data":"81065a440959fbab0d61e229ceebfa00a12256a809edbb90b591870a7a7992bf"}
Jul 16 17:43:37.111570 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.111522    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:37.129989 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.129950    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-fd65d7d6-7xg4w" podStartSLOduration=1.506719276 podStartE2EDuration="5.129936746s" totalImagesPullingTime="3.62321747s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:37.127405516 +0000 UTC m=+547.551135591" watchObservedRunningTime="2026-07-16 17:43:37.129936746 +0000 UTC m=+547.553666805"
Jul 16 17:43:37.151479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:37.151413    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-84b54b7f97-knwdl" podStartSLOduration=0.987385497 podStartE2EDuration="5.151399078s" totalImagesPullingTime="4.164013581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:43:37.150092211 +0000 UTC m=+547.573822272" watchObservedRunningTime="2026-07-16 17:43:37.151399078 +0000 UTC m=+547.575129172"
Jul 16 17:43:48.118100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:48.118074    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-84b54b7f97-knwdl"
Jul 16 17:43:48.120012 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:43:48.119996    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-fd65d7d6-7xg4w"
Jul 16 17:44:29.945782 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:44:29.945753    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:44:29.946260 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:44:29.945900    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:45:54.813673 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.813641    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"]
Jul 16 17:45:54.816627 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.816611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.819321 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.819292    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving\""
Jul 16 17:45:54.819426 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.819326    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 16 17:45:54.824494 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.824471    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"]
Jul 16 17:45:54.837269 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.837245    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/940bd3d7-fbda-4af0-b68c-4e96c9c21017-data\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.837386 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.837277    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.837386 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.837353    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwd7\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-kube-api-access-lbwd7\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.938089 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.938054    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwd7\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-kube-api-access-lbwd7\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.938261 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.938104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/940bd3d7-fbda-4af0-b68c-4e96c9c21017-data\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.938261 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.938124    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.938482 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.938461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/940bd3d7-fbda-4af0-b68c-4e96c9c21017-data\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.940585 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.940559    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"seaweedfs-tls-serving\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-seaweedfs-tls-serving\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:54.947616 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:54.947598    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwd7\" (UniqueName: \"kubernetes.io/projected/940bd3d7-fbda-4af0-b68c-4e96c9c21017-kube-api-access-lbwd7\") pod \"seaweedfs-tls-serving-67b7477c6d-w8fc9\" (UID: \"940bd3d7-fbda-4af0-b68c-4e96c9c21017\") " pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:55.125584 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:55.125502    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"
Jul 16 17:45:55.252561 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:55.252530    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9"]
Jul 16 17:45:55.255637 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:45:55.255613    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940bd3d7_fbda_4af0_b68c_4e96c9c21017.slice/crio-958049a366069c8c56a8a91d8ab0adf7b8babb44a54bccd4638b30c33b630109 WatchSource:0}: Error finding container 958049a366069c8c56a8a91d8ab0adf7b8babb44a54bccd4638b30c33b630109: Status 404 returned error can't find the container with id 958049a366069c8c56a8a91d8ab0adf7b8babb44a54bccd4638b30c33b630109
Jul 16 17:45:55.610268 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:55.610230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9" event={"ID":"940bd3d7-fbda-4af0-b68c-4e96c9c21017","Type":"ContainerStarted","Data":"8e4463f4538b980e07ce664234ed59fe0c26253a8622608411071ece528eecb5"}
Jul 16 17:45:55.610412 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:55.610290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9" event={"ID":"940bd3d7-fbda-4af0-b68c-4e96c9c21017","Type":"ContainerStarted","Data":"958049a366069c8c56a8a91d8ab0adf7b8babb44a54bccd4638b30c33b630109"}
Jul 16 17:45:55.627530 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:45:55.627487    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-tls-serving-67b7477c6d-w8fc9" podStartSLOduration=1.350961101 podStartE2EDuration="1.627470819s" totalImagesPullingTime="276.509718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:45:55.626737224 +0000 UTC m=+686.050467281" watchObservedRunningTime="2026-07-16 17:45:55.627470819 +0000 UTC m=+686.051200879"
Jul 16 17:46:18.126824 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.126791    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:46:18.131031 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.131006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.133541 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133519    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:46:18.133662 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133522    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:46:18.133662 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:46:18.133813 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133798    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 16 17:46:18.133887 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133840    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-52dqs\""
Jul 16 17:46:18.133887 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 16 17:46:18.133980 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.133913    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:46:18.144429 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.144410    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:46:18.211693 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.211665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.211853 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.211699    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.211853 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.211809    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.211927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.211858    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.211927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.211889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r688j\" (UniqueName: \"kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312634 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312634 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312552    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312634 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312634 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312600    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r688j\" (UniqueName: \"kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.312818 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:46:18.312724    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-serving-cert: secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 16 17:46:18.312818 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:46:18.312790    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls podName:e598e9d6-b79f-44e1-acb2-4fdf30937a87 nodeName:}" failed. No retries permitted until 2026-07-16 17:46:18.812771546 +0000 UTC m=+709.236501592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls") pod "isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87") : secret "isvc-sklearn-batcher-predictor-serving-cert" not found
Jul 16 17:46:18.312925 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.312902    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.313296 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.313277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.313333 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.313277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.321969 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.321942    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r688j\" (UniqueName: \"kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.816571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.816537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:18.819132 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:18.819100    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-85c9b85988-7wtj9\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:19.042172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:19.042135    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:19.178561 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:19.178530    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:46:19.181452 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:46:19.181404    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode598e9d6_b79f_44e1_acb2_4fdf30937a87.slice/crio-baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20 WatchSource:0}: Error finding container baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20: Status 404 returned error can't find the container with id baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20
Jul 16 17:46:19.183492 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:19.183477    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:46:19.707787 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:19.707735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerStarted","Data":"baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20"}
Jul 16 17:46:22.722934 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:22.722849    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerStarted","Data":"f414d4d8e7f690e8237b13fda150cd9bff8d252c2084cf74ba6785e8e70528f4"}
Jul 16 17:46:26.739303 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:26.739266    2576 generic.go:356] "Generic (PLEG): container finished" podID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerID="f414d4d8e7f690e8237b13fda150cd9bff8d252c2084cf74ba6785e8e70528f4" exitCode=0
Jul 16 17:46:26.739694 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:26.739312    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerDied","Data":"f414d4d8e7f690e8237b13fda150cd9bff8d252c2084cf74ba6785e8e70528f4"}
Jul 16 17:46:40.811200 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:40.811167    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerStarted","Data":"013011bdf2cfd9630ab7779e8547b6f948e41fb58d955096a4d4a721e5d29518"}
Jul 16 17:46:42.822757 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:42.822721    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerStarted","Data":"fee343f3206d025617d8169d2f658ac97d6102fde89ee9f66502dfc88b077a59"}
Jul 16 17:46:45.837997 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:45.837958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerStarted","Data":"8c5aed63385b8847ecf45f50c9972309f664536cf0785f89cb4c3ef526673066"}
Jul 16 17:46:45.838415 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:45.838031    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:45.838415 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:45.838200    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:45.839659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:45.839633    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:46:45.859156 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:45.859118    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podStartSLOduration=2.814200057 podStartE2EDuration="27.859085475s" totalImagesPullingTime="21.518605352s" totalInitContainerRuntime="3.526280066s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:46:18 +0000 UTC" imagePullSessionsCount=4 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:46:45.858177437 +0000 UTC m=+736.281907537" watchObservedRunningTime="2026-07-16 17:46:45.859085475 +0000 UTC m=+736.282815517"
Jul 16 17:46:46.842248 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:46.842206    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:46.842743 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:46.842290    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:46:46.843487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:46.843458    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:47.846478 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:47.846415    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:46:47.847040 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:47.847012    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:47.855401 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:47.855374    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:46:48.850326 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:48.850291    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:46:48.850721 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:48.850637    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:46:58.850907 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:58.850858    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:46:58.851368 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:46:58.851342    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:08.850839 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:08.850789    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:47:08.851388 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:08.851254    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:18.850475 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:18.850406    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:47:18.850917 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:18.850885    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:28.851195 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:28.851143    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:47:28.851755 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:28.851687    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:38.850938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:38.850893    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:47:38.851490 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:38.851323    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:48.850778 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:48.850729    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:47:48.851248 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:48.851180    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:47:58.850765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:58.850667    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:47:58.851323 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:47:58.851242    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:13.139714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.139679    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:48:13.140216 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.140011    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" containerID="cri-o://013011bdf2cfd9630ab7779e8547b6f948e41fb58d955096a4d4a721e5d29518" gracePeriod=30
Jul 16 17:48:13.140216 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.140082    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" containerID="cri-o://fee343f3206d025617d8169d2f658ac97d6102fde89ee9f66502dfc88b077a59" gracePeriod=30
Jul 16 17:48:13.140216 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.140045    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" containerID="cri-o://8c5aed63385b8847ecf45f50c9972309f664536cf0785f89cb4c3ef526673066" gracePeriod=30
Jul 16 17:48:13.398130 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.398044    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:48:13.403190 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.403169    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.405853 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.405829    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 16 17:48:13.405980 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.405831    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 16 17:48:13.411145 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.411123    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:48:13.476311 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.476281    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.476474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.476339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.476474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.476395    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.476474 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.476422    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.476583 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.476478    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz6rf\" (UniqueName: \"kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577290 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577317    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577339    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz6rf\" (UniqueName: \"kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.577683 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:48:13.577536    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-serving-cert: secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 16 17:48:13.577683 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:48:13.577612    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls podName:2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b nodeName:}" failed. No retries permitted until 2026-07-16 17:48:14.077592533 +0000 UTC m=+824.501322575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls") pod "isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b") : secret "isvc-sklearn-batcher-custom-predictor-serving-cert" not found
Jul 16 17:48:13.577822 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.577798    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.578083 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.578065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.578126 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.578068    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:13.587028 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:13.587007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz6rf\" (UniqueName: \"kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:14.082097 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.082059    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:14.084599 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.084577    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:14.172153 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.172123    2576 generic.go:356] "Generic (PLEG): container finished" podID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerID="fee343f3206d025617d8169d2f658ac97d6102fde89ee9f66502dfc88b077a59" exitCode=2
Jul 16 17:48:14.172528 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.172171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerDied","Data":"fee343f3206d025617d8169d2f658ac97d6102fde89ee9f66502dfc88b077a59"}
Jul 16 17:48:14.314334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.314299    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:14.450702 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:14.450679    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:48:14.453973 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:48:14.453944    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0ba9be_2ee8_4c34_a98c_bf52acfbb45b.slice/crio-e0bdb6f757ac7ee8cde27861de5e591191b2eb3059ebeee73485a29b610f3afd WatchSource:0}: Error finding container e0bdb6f757ac7ee8cde27861de5e591191b2eb3059ebeee73485a29b610f3afd: Status 404 returned error can't find the container with id e0bdb6f757ac7ee8cde27861de5e591191b2eb3059ebeee73485a29b610f3afd
Jul 16 17:48:15.176962 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:15.176926    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerStarted","Data":"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"}
Jul 16 17:48:15.177364 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:15.176988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerStarted","Data":"e0bdb6f757ac7ee8cde27861de5e591191b2eb3059ebeee73485a29b610f3afd"}
Jul 16 17:48:17.846796 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:17.846712    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:18.190881 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.190851    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerID="7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23" exitCode=0
Jul 16 17:48:18.191016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.190923    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerDied","Data":"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"}
Jul 16 17:48:18.193477 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.193450    2576 generic.go:356] "Generic (PLEG): container finished" podID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerID="013011bdf2cfd9630ab7779e8547b6f948e41fb58d955096a4d4a721e5d29518" exitCode=0
Jul 16 17:48:18.193477 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.193469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerDied","Data":"013011bdf2cfd9630ab7779e8547b6f948e41fb58d955096a4d4a721e5d29518"}
Jul 16 17:48:18.850826 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.850790    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:48:18.851804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:18.851772    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:19.200037 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.199947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerStarted","Data":"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"}
Jul 16 17:48:19.200172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.200049    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerStarted","Data":"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"}
Jul 16 17:48:19.200172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.200075    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerStarted","Data":"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"}
Jul 16 17:48:19.200377 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.200359    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:19.200522 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.200504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:19.202067 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.202041    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:19.223601 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:19.223559    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podStartSLOduration=2.5929508119999998 podStartE2EDuration="6.223543638s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.630592826s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:48:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:48:19.22191904 +0000 UTC m=+829.645649127" watchObservedRunningTime="2026-07-16 17:48:19.223543638 +0000 UTC m=+829.647273698"
Jul 16 17:48:20.204832 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:20.204803    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:20.205214 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:20.204839    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:20.206088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:20.206062    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:21.208927 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:21.208888    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:21.209363 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:21.209288    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:22.847779 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:22.847734    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:26.213583 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:26.213554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:48:26.214183 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:26.214151    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:26.214521 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:26.214492    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:27.846836 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:27.846795    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:27.847207 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:27.846938    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:28.850919 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:28.850878    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:48:28.852494 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:28.852467    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:32.847536 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:32.847499    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:36.214094 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:36.214045    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:36.214500 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:36.214469    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:37.847379 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:37.847330    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:38.850935 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:38.850892    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 16 17:48:38.851279 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:38.851048    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:38.851693 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:38.851670    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:38.851819 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:38.851806    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:42.847182 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:42.847136    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 16 17:48:43.296931 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.296903    2576 generic.go:356] "Generic (PLEG): container finished" podID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerID="8c5aed63385b8847ecf45f50c9972309f664536cf0785f89cb4c3ef526673066" exitCode=0
Jul 16 17:48:43.297073 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.296959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerDied","Data":"8c5aed63385b8847ecf45f50c9972309f664536cf0785f89cb4c3ef526673066"}
Jul 16 17:48:43.297073 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.297025    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9" event={"ID":"e598e9d6-b79f-44e1-acb2-4fdf30937a87","Type":"ContainerDied","Data":"baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20"}
Jul 16 17:48:43.297073 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.297061    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="baa01c8f5c35878c5c17d64a0ce4e00286040009d6396f7e011df289155f9e20"
Jul 16 17:48:43.300053 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.300033    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:43.427263 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427171    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config\") pod \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") "
Jul 16 17:48:43.427263 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location\") pod \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") "
Jul 16 17:48:43.427263 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427240    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert\") pod \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") "
Jul 16 17:48:43.427548 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427276    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r688j\" (UniqueName: \"kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j\") pod \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") "
Jul 16 17:48:43.427548 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427323    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") pod \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\" (UID: \"e598e9d6-b79f-44e1-acb2-4fdf30937a87\") "
Jul 16 17:48:43.427659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427614    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "e598e9d6-b79f-44e1-acb2-4fdf30937a87" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:48:43.427706 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e598e9d6-b79f-44e1-acb2-4fdf30937a87" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:43.427706 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.427644    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "e598e9d6-b79f-44e1-acb2-4fdf30937a87" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:48:43.429711 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.429682    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e598e9d6-b79f-44e1-acb2-4fdf30937a87" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:48:43.429806 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.429763    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j" (OuterVolumeSpecName: "kube-api-access-r688j") pod "e598e9d6-b79f-44e1-acb2-4fdf30937a87" (UID: "e598e9d6-b79f-44e1-acb2-4fdf30937a87"). InnerVolumeSpecName "kube-api-access-r688j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:48:43.528500 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.528462    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:48:43.528500 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.528492    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:48:43.528500 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.528502    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/e598e9d6-b79f-44e1-acb2-4fdf30937a87-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:48:43.528746 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.528511    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r688j\" (UniqueName: \"kubernetes.io/projected/e598e9d6-b79f-44e1-acb2-4fdf30937a87-kube-api-access-r688j\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:48:43.528746 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:43.528522    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e598e9d6-b79f-44e1-acb2-4fdf30937a87-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:48:44.300869 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:44.300840    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"
Jul 16 17:48:44.321374 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:44.321345    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:48:44.326082 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:44.326061    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-85c9b85988-7wtj9"]
Jul 16 17:48:46.090965 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:46.090930    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" path="/var/lib/kubelet/pods/e598e9d6-b79f-44e1-acb2-4fdf30937a87/volumes"
Jul 16 17:48:46.214098 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:46.214058    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:46.214562 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:46.214538    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:48:56.214675 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:56.214625    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:48:56.215079 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:48:56.215052    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:49:06.214255 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:06.214213    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:49:06.214765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:06.214632    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:49:16.214159 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:16.214119    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:49:16.214747 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:16.214644    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:49:26.214633 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:26.214544    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:49:26.215152 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:26.215118    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:49:29.970161 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:29.970130    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:49:29.973002 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:29.972980    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:49:36.214526 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:36.214494    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:49:36.215015 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:36.214903    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:49:48.438699 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.438658    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:49:48.439255 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.438978    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" containerID="cri-o://91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d" gracePeriod=30
Jul 16 17:49:48.439255 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.439017    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" containerID="cri-o://6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27" gracePeriod=30
Jul 16 17:49:48.439255 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.439032    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" containerID="cri-o://1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b" gracePeriod=30
Jul 16 17:49:48.521459 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521408    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:49:48.521763 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521749    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy"
Jul 16 17:49:48.521804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521765    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy"
Jul 16 17:49:48.521804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521777    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent"
Jul 16 17:49:48.521804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521782    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent"
Jul 16 17:49:48.521804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="storage-initializer"
Jul 16 17:49:48.521804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521798    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="storage-initializer"
Jul 16 17:49:48.521958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521814    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container"
Jul 16 17:49:48.521958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521820    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container"
Jul 16 17:49:48.521958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521865    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="agent"
Jul 16 17:49:48.521958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521874    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kube-rbac-proxy"
Jul 16 17:49:48.521958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.521881    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e598e9d6-b79f-44e1-acb2-4fdf30937a87" containerName="kserve-container"
Jul 16 17:49:48.525158 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.525138    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.527225 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.527207    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 16 17:49:48.527329 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.527255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 16 17:49:48.533877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.533854    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:49:48.557171 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.557148    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerID="1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b" exitCode=2
Jul 16 17:49:48.557290 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.557193    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerDied","Data":"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"}
Jul 16 17:49:48.561624 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.561598    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2qpl\" (UniqueName: \"kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.561726 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.561656    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.561726 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.561682    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.662057 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.662021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2qpl\" (UniqueName: \"kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.662244 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.662067    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.662244 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.662087    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.662733 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.662711    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.664604 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.664584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.670323 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.670301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2qpl\" (UniqueName: \"kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl\") pod \"message-dumper-predictor-6575bcfdfd-mlvsd\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.835638 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.835609    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:48.967862 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:48.967833    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:49:48.971534 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:49:48.971506    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba16e4f_7f77_4b1d_a95a_a3280f4230d1.slice/crio-b150523b294c57e0c4609902acacf3a0372d12034c5af409c505bb83c4a94e7b WatchSource:0}: Error finding container b150523b294c57e0c4609902acacf3a0372d12034c5af409c505bb83c4a94e7b: Status 404 returned error can't find the container with id b150523b294c57e0c4609902acacf3a0372d12034c5af409c505bb83c4a94e7b
Jul 16 17:49:49.561990 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:49.561949    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerStarted","Data":"b150523b294c57e0c4609902acacf3a0372d12034c5af409c505bb83c4a94e7b"}
Jul 16 17:49:50.567396 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.567357    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerStarted","Data":"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"}
Jul 16 17:49:50.567843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.567427    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerStarted","Data":"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"}
Jul 16 17:49:50.567843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.567584    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:50.567843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.567629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:50.569554 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.569539    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:50.585271 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:50.585234    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" podStartSLOduration=1.328885971 podStartE2EDuration="2.585221971s" totalImagesPullingTime="1.256336s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:49:50.584868546 +0000 UTC m=+921.008598607" watchObservedRunningTime="2026-07-16 17:49:50.585221971 +0000 UTC m=+921.008952031"
Jul 16 17:49:51.209528 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:51.209489    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:49:53.583495 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:53.583465    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerID="91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d" exitCode=0
Jul 16 17:49:53.583926 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:53.583537    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerDied","Data":"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"}
Jul 16 17:49:56.209809 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:56.209762    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:49:56.214513 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:56.214481    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:49:56.215724 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:56.215701    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:49:57.580110 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:57.580079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:49:58.568278 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.568244    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:49:58.571965 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.571949    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.574213 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.574186    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 16 17:49:58.574335 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.574224    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 16 17:49:58.582938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.582916    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:49:58.639960 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.639934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxgnx\" (UniqueName: \"kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.640100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.639977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.640100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.640009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.640100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.640037    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.640100 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.640085    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741078 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741046    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741240 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741085    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741240 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741112    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741240 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741151    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741240 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741186    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxgnx\" (UniqueName: \"kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741609 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741847 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741827    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.741911 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.741831    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.743791 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.743775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.749862 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.749839    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxgnx\" (UniqueName: \"kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx\") pod \"isvc-logger-predictor-c6f84955-2krt2\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:58.882083 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:58.881998    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:49:59.015551 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:59.015518    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:49:59.018420 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:49:59.018397    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b3f2e77_9ed5_4dd7_8aab_87f5373b9bd4.slice/crio-88fd372a23b71bfd50cb14310375a0ca7f6b8867c65473b3368151d8e41b21ec WatchSource:0}: Error finding container 88fd372a23b71bfd50cb14310375a0ca7f6b8867c65473b3368151d8e41b21ec: Status 404 returned error can't find the container with id 88fd372a23b71bfd50cb14310375a0ca7f6b8867c65473b3368151d8e41b21ec
Jul 16 17:49:59.610227 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:59.610191    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerStarted","Data":"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"}
Jul 16 17:49:59.610622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:49:59.610282    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerStarted","Data":"88fd372a23b71bfd50cb14310375a0ca7f6b8867c65473b3368151d8e41b21ec"}
Jul 16 17:50:01.209809 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:01.209767    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:50:01.210229 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:01.209918    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:50:03.628897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:03.628864    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerID="1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0" exitCode=0
Jul 16 17:50:03.629275 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:03.628943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerDied","Data":"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"}
Jul 16 17:50:04.641521 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.641485    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerStarted","Data":"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"}
Jul 16 17:50:04.641897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.641579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerStarted","Data":"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"}
Jul 16 17:50:04.641897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.641605    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerStarted","Data":"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"}
Jul 16 17:50:04.641897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.641883    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:50:04.642049 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.642029    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:50:04.643396 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.643369    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:04.664476 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:04.664410    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podStartSLOduration=2.786826114 podStartE2EDuration="6.664396023s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.877569909s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:50:04.662013303 +0000 UTC m=+935.085743367" watchObservedRunningTime="2026-07-16 17:50:04.664396023 +0000 UTC m=+935.088126083"
Jul 16 17:50:05.646969 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:05.646940    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:50:05.647375 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:05.647144    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:05.648260 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:05.648233    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:06.209837 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:06.209802    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:50:06.214472 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:06.214424    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:06.215637 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:06.215617    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:50:06.651902 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:06.651863    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:06.652334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:06.652185    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:11.209621 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:11.209571    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:50:11.656402 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:11.656373    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:50:11.657077 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:11.657042    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:11.657306 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:11.657274    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:16.209806 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:16.209760    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 16 17:50:16.214342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:16.214312    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:16.214471 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:16.214459    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:50:16.215619 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:16.215594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:5000: connect: connection refused"
Jul 16 17:50:16.215730 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:16.215703    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:50:18.591936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.591913    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:50:18.703918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.703839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert\") pod \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") "
Jul 16 17:50:18.703918 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.703880    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location\") pod \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") "
Jul 16 17:50:18.704141 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.703936    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config\") pod \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") "
Jul 16 17:50:18.704141 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.703995    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") pod \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") "
Jul 16 17:50:18.704141 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.704023    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz6rf\" (UniqueName: \"kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf\") pod \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\" (UID: \"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b\") "
Jul 16 17:50:18.704266 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.704184    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:50:18.704342 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.704323    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:50:18.704406 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.704322    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:50:18.704509 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.704489    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:50:18.706372 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.706344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf" (OuterVolumeSpecName: "kube-api-access-jz6rf") pod "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b"). InnerVolumeSpecName "kube-api-access-jz6rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:50:18.706487 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.706451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" (UID: "2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:50:18.707787 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.707763    2576 generic.go:356] "Generic (PLEG): container finished" podID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerID="6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27" exitCode=0
Jul 16 17:50:18.707869 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.707839    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"
Jul 16 17:50:18.707869 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.707843    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerDied","Data":"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"}
Jul 16 17:50:18.707943 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.707910    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf" event={"ID":"2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b","Type":"ContainerDied","Data":"e0bdb6f757ac7ee8cde27861de5e591191b2eb3059ebeee73485a29b610f3afd"}
Jul 16 17:50:18.707983 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.707944    2576 scope.go:117] "RemoveContainer" containerID="6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"
Jul 16 17:50:18.724554 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.724534    2576 scope.go:117] "RemoveContainer" containerID="1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"
Jul 16 17:50:18.731804 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.731789    2576 scope.go:117] "RemoveContainer" containerID="91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"
Jul 16 17:50:18.737292 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.737266    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:50:18.739903 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.739883    2576 scope.go:117] "RemoveContainer" containerID="7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"
Jul 16 17:50:18.741658 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.741640    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-59cbb4c5bc-r96gf"]
Jul 16 17:50:18.747059 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747044    2576 scope.go:117] "RemoveContainer" containerID="6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"
Jul 16 17:50:18.747339 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:50:18.747316    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27\": container with ID starting with 6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27 not found: ID does not exist" containerID="6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"
Jul 16 17:50:18.747391 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747342    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27"} err="failed to get container status \"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27\": rpc error: code = NotFound desc = could not find container \"6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27\": container with ID starting with 6da302de403f9cb1b440d5b1dcd728dd177e66e742e638e73458104440097c27 not found: ID does not exist"
Jul 16 17:50:18.747391 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747359    2576 scope.go:117] "RemoveContainer" containerID="1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"
Jul 16 17:50:18.747629 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:50:18.747611    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b\": container with ID starting with 1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b not found: ID does not exist" containerID="1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"
Jul 16 17:50:18.747724 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747632    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b"} err="failed to get container status \"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b\": rpc error: code = NotFound desc = could not find container \"1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b\": container with ID starting with 1b7dc551b956a087ae6fda69bd32f93863ca52f7cdc1a5dd8dfc736236ced92b not found: ID does not exist"
Jul 16 17:50:18.747724 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747647    2576 scope.go:117] "RemoveContainer" containerID="91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"
Jul 16 17:50:18.747870 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:50:18.747855    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d\": container with ID starting with 91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d not found: ID does not exist" containerID="91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"
Jul 16 17:50:18.747909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d"} err="failed to get container status \"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d\": rpc error: code = NotFound desc = could not find container \"91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d\": container with ID starting with 91dd8606630818fd604c26b282c7f0304a46eb6efa0296681d0e5fd67da8cd1d not found: ID does not exist"
Jul 16 17:50:18.747909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.747886    2576 scope.go:117] "RemoveContainer" containerID="7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"
Jul 16 17:50:18.748096 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:50:18.748081    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23\": container with ID starting with 7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23 not found: ID does not exist" containerID="7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"
Jul 16 17:50:18.748134 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.748102    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23"} err="failed to get container status \"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23\": rpc error: code = NotFound desc = could not find container \"7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23\": container with ID starting with 7c2afb9e68174adaf87dcbface9dd15075a0d35ab01457b050f48eedcf5f3e23 not found: ID does not exist"
Jul 16 17:50:18.804769 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.804735    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:50:18.804769 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.804762    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:50:18.804769 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.804772    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:50:18.804978 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:18.804783    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz6rf\" (UniqueName: \"kubernetes.io/projected/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b-kube-api-access-jz6rf\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:50:20.091257 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:20.091226    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" path="/var/lib/kubelet/pods/2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b/volumes"
Jul 16 17:50:21.657182 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:21.657143    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:21.657645 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:21.657622    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:31.657469 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:31.657414    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:31.657958 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:31.657934    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:41.657638 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:41.657594    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:41.658121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:41.658098    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:50:51.657302 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:51.657204    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:50:51.657784 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:50:51.657708    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:01.657131 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:01.657089    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:51:01.657604 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:01.657554    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:11.657168 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:11.657120    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:51:11.657678 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:11.657584    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:21.658228 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:21.658198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:51:21.658687 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:21.658669    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:51:33.585892 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.585859    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-mlvsd_3ba16e4f-7f77-4b1d-a95a-a3280f4230d1/kserve-container/0.log"
Jul 16 17:51:33.871528 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.871427    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:51:33.871932 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.871903    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" containerID="cri-o://c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761" gracePeriod=30
Jul 16 17:51:33.872302 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.872246    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" containerID="cri-o://785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639" gracePeriod=30
Jul 16 17:51:33.874505 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.872305    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" containerID="cri-o://fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158" gracePeriod=30
Jul 16 17:51:33.943186 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943145    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:51:33.943654 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943638    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="storage-initializer"
Jul 16 17:51:33.943717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943658    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="storage-initializer"
Jul 16 17:51:33.943717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943676    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container"
Jul 16 17:51:33.943717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943684    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container"
Jul 16 17:51:33.943717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943699    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent"
Jul 16 17:51:33.943717 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943708    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent"
Jul 16 17:51:33.943877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943727    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy"
Jul 16 17:51:33.943877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943736    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy"
Jul 16 17:51:33.943877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943815    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kserve-container"
Jul 16 17:51:33.943877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943829    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="kube-rbac-proxy"
Jul 16 17:51:33.943877 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.943841    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c0ba9be-2ee8-4c34-a98c-bf52acfbb45b" containerName="agent"
Jul 16 17:51:33.947551 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.947517    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:33.950963 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.950936    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 16 17:51:33.951157 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.951137    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 16 17:51:33.961914 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.961886    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:51:33.982306 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.982280    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:51:33.982558 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.982518    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kserve-container" containerID="cri-o://a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08" gracePeriod=30
Jul 16 17:51:33.982614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:33.982591    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kube-rbac-proxy" containerID="cri-o://1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798" gracePeriod=30
Jul 16 17:51:34.025830 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.025802    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerID="fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158" exitCode=2
Jul 16 17:51:34.025953 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.025882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerDied","Data":"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"}
Jul 16 17:51:34.123550 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.123453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.123550 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.123508    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.123727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.123564    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.123727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.123595    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.123727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.123638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqssq\" (UniqueName: \"kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.224719 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.224684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.224844 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.224736    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.224844 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.224767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqssq\" (UniqueName: \"kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.224844 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.224814    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.224844 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.224838    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.225252 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.225231    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.225493 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.225461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.225546 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.225516    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.227286 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.227270    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.235720 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.235698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqssq\" (UniqueName: \"kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.240223 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.240206    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:51:34.261333 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.261307    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:51:34.398297 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.398273    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:51:34.402475 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:51:34.402400    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19284549_063a_45a1_b4bd_aad87f132108.slice/crio-59883203bfc882a7f3a746cd0049430631cf4ac5cdd127f3aedbbc243462e78f WatchSource:0}: Error finding container 59883203bfc882a7f3a746cd0049430631cf4ac5cdd127f3aedbbc243462e78f: Status 404 returned error can't find the container with id 59883203bfc882a7f3a746cd0049430631cf4ac5cdd127f3aedbbc243462e78f
Jul 16 17:51:34.426373 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.426350    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2qpl\" (UniqueName: \"kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl\") pod \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") "
Jul 16 17:51:34.426488 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.426382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls\") pod \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") "
Jul 16 17:51:34.426488 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.426406    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config\") pod \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\" (UID: \"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1\") "
Jul 16 17:51:34.426714 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.426693    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" (UID: "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:51:34.428453 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.428406    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl" (OuterVolumeSpecName: "kube-api-access-h2qpl") pod "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" (UID: "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1"). InnerVolumeSpecName "kube-api-access-h2qpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:51:34.428547 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.428473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" (UID: "3ba16e4f-7f77-4b1d-a95a-a3280f4230d1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:51:34.527646 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.527604    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2qpl\" (UniqueName: \"kubernetes.io/projected/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-api-access-h2qpl\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:51:34.527646 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.527637    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:51:34.527646 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:34.527652    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:51:35.032121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032086    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerID="1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798" exitCode=2
Jul 16 17:51:35.032121 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032114    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerID="a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08" exitCode=2
Jul 16 17:51:35.032571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032134    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"
Jul 16 17:51:35.032571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032171    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerDied","Data":"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"}
Jul 16 17:51:35.032571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032245    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerDied","Data":"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"}
Jul 16 17:51:35.032571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032276    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd" event={"ID":"3ba16e4f-7f77-4b1d-a95a-a3280f4230d1","Type":"ContainerDied","Data":"b150523b294c57e0c4609902acacf3a0372d12034c5af409c505bb83c4a94e7b"}
Jul 16 17:51:35.032571 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.032311    2576 scope.go:117] "RemoveContainer" containerID="1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"
Jul 16 17:51:35.033931 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.033893    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerStarted","Data":"092a632d0b3bbebff8e898524672085fba710d9237272195769c71bfe7502bd0"}
Jul 16 17:51:35.034021 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.033971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerStarted","Data":"59883203bfc882a7f3a746cd0049430631cf4ac5cdd127f3aedbbc243462e78f"}
Jul 16 17:51:35.041070 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.041053    2576 scope.go:117] "RemoveContainer" containerID="a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"
Jul 16 17:51:35.048900 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.048884    2576 scope.go:117] "RemoveContainer" containerID="1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"
Jul 16 17:51:35.049131 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:51:35.049116    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798\": container with ID starting with 1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798 not found: ID does not exist" containerID="1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"
Jul 16 17:51:35.049176 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049138    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"} err="failed to get container status \"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798\": rpc error: code = NotFound desc = could not find container \"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798\": container with ID starting with 1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798 not found: ID does not exist"
Jul 16 17:51:35.049176 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049153    2576 scope.go:117] "RemoveContainer" containerID="a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"
Jul 16 17:51:35.049388 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:51:35.049371    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08\": container with ID starting with a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08 not found: ID does not exist" containerID="a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"
Jul 16 17:51:35.049424 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049393    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"} err="failed to get container status \"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08\": rpc error: code = NotFound desc = could not find container \"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08\": container with ID starting with a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08 not found: ID does not exist"
Jul 16 17:51:35.049424 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049408    2576 scope.go:117] "RemoveContainer" containerID="1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"
Jul 16 17:51:35.049630 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049614    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798"} err="failed to get container status \"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798\": rpc error: code = NotFound desc = could not find container \"1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798\": container with ID starting with 1783c142225b4e9725120e501eb120ee86cead4e5bd566765430e9546567d798 not found: ID does not exist"
Jul 16 17:51:35.049671 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049631    2576 scope.go:117] "RemoveContainer" containerID="a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"
Jul 16 17:51:35.049841 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.049827    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08"} err="failed to get container status \"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08\": rpc error: code = NotFound desc = could not find container \"a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08\": container with ID starting with a3f6d673ab2beb7f337fb1e57e54f3eda38726daa55e169916e42389f74b1e08 not found: ID does not exist"
Jul 16 17:51:35.071071 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.071052    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:51:35.075549 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:35.075529    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-mlvsd"]
Jul 16 17:51:36.090944 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:36.090911    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" path="/var/lib/kubelet/pods/3ba16e4f-7f77-4b1d-a95a-a3280f4230d1/volumes"
Jul 16 17:51:36.652024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:36.651982    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:51:39.054073 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:39.054033    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerID="c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761" exitCode=0
Jul 16 17:51:39.054461 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:39.054110    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerDied","Data":"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"}
Jul 16 17:51:41.652978 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:41.652937    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:51:41.658062 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:41.658023    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:41.659254 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:41.659232    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:51:46.652307 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:46.652257    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:51:46.652765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:46.652396    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:51:51.652833 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:51.652780    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:51:51.657544 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:51.657512    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:51:51.658655 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:51.658629    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:51:56.652705 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:51:56.652655    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:52:01.652063 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:01.652026    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 16 17:52:01.657852 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:01.657828    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 17:52:01.657936 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:01.657927    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:52:01.659012 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:01.658985    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 16 17:52:01.659122 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:01.659072    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:52:04.020456 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.020407    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:52:04.069396 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069371    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls\") pod \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") "
Jul 16 17:52:04.069543 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069478    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location\") pod \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") "
Jul 16 17:52:04.069664 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069641    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert\") pod \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") "
Jul 16 17:52:04.069727 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069712    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxgnx\" (UniqueName: \"kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx\") pod \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") "
Jul 16 17:52:04.069765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069724    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" (UID: "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:52:04.069765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config\") pod \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\" (UID: \"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4\") "
Jul 16 17:52:04.069979 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.069961    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:52:04.070055 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.070034    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" (UID: "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:52:04.070211 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.070070    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" (UID: "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:52:04.071614 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.071596    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" (UID: "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:52:04.071824 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.071807    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx" (OuterVolumeSpecName: "kube-api-access-xxgnx") pod "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" (UID: "6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4"). InnerVolumeSpecName "kube-api-access-xxgnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:52:04.165367 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.165291    2576 generic.go:356] "Generic (PLEG): container finished" podID="19284549-063a-45a1-b4bd-aad87f132108" containerID="092a632d0b3bbebff8e898524672085fba710d9237272195769c71bfe7502bd0" exitCode=0
Jul 16 17:52:04.165533 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.165367    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerDied","Data":"092a632d0b3bbebff8e898524672085fba710d9237272195769c71bfe7502bd0"}
Jul 16 17:52:04.166909 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.166891    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 17:52:04.167921 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.167900    2576 generic.go:356] "Generic (PLEG): container finished" podID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerID="785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639" exitCode=0
Jul 16 17:52:04.167987 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.167954    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"
Jul 16 17:52:04.168022 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.167979    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerDied","Data":"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"}
Jul 16 17:52:04.168054 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.168044    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2" event={"ID":"6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4","Type":"ContainerDied","Data":"88fd372a23b71bfd50cb14310375a0ca7f6b8867c65473b3368151d8e41b21ec"}
Jul 16 17:52:04.168103 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.168091    2576 scope.go:117] "RemoveContainer" containerID="785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"
Jul 16 17:52:04.170574 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.170557    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:52:04.170574 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.170578    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxgnx\" (UniqueName: \"kubernetes.io/projected/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-api-access-xxgnx\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:52:04.170704 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.170595    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:52:04.170704 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.170608    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:52:04.176160 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.176145    2576 scope.go:117] "RemoveContainer" containerID="fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"
Jul 16 17:52:04.183784 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.183768    2576 scope.go:117] "RemoveContainer" containerID="c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"
Jul 16 17:52:04.191765 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.191747    2576 scope.go:117] "RemoveContainer" containerID="1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"
Jul 16 17:52:04.198610 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.198593    2576 scope.go:117] "RemoveContainer" containerID="785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"
Jul 16 17:52:04.198861 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:52:04.198844    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639\": container with ID starting with 785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639 not found: ID does not exist" containerID="785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"
Jul 16 17:52:04.198919 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.198873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639"} err="failed to get container status \"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639\": rpc error: code = NotFound desc = could not find container \"785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639\": container with ID starting with 785bc1446b6abf2624a463003febcaac33fe8d3ceb3511d82f673718a77e3639 not found: ID does not exist"
Jul 16 17:52:04.198919 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.198892    2576 scope.go:117] "RemoveContainer" containerID="fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"
Jul 16 17:52:04.199243 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:52:04.199216    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158\": container with ID starting with fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158 not found: ID does not exist" containerID="fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"
Jul 16 17:52:04.199376 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.199255    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158"} err="failed to get container status \"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158\": rpc error: code = NotFound desc = could not find container \"fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158\": container with ID starting with fa0524b9de317c0cad1a61b15295305f053f364c8a244fe734bfb461465e6158 not found: ID does not exist"
Jul 16 17:52:04.199376 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.199278    2576 scope.go:117] "RemoveContainer" containerID="c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"
Jul 16 17:52:04.199716 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:52:04.199676    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761\": container with ID starting with c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761 not found: ID does not exist" containerID="c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"
Jul 16 17:52:04.199835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.199724    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761"} err="failed to get container status \"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761\": rpc error: code = NotFound desc = could not find container \"c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761\": container with ID starting with c1adb105521c629ff56f7fea5a4d05717c01b1a81c53e45dfaa9874d8521b761 not found: ID does not exist"
Jul 16 17:52:04.199835 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.199746    2576 scope.go:117] "RemoveContainer" containerID="1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"
Jul 16 17:52:04.200100 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:52:04.200080    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0\": container with ID starting with 1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0 not found: ID does not exist" containerID="1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"
Jul 16 17:52:04.200168 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.200104    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0"} err="failed to get container status \"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0\": rpc error: code = NotFound desc = could not find container \"1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0\": container with ID starting with 1f7ff93b5cd700f237f67503e54827c2e951c39582a1a049cd8c8bc651a575b0 not found: ID does not exist"
Jul 16 17:52:04.201623 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.201606    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:52:04.206180 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:04.206161    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-c6f84955-2krt2"]
Jul 16 17:52:06.093228 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:06.093196    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" path="/var/lib/kubelet/pods/6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4/volumes"
Jul 16 17:52:30.010713 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:52:30.010631    2576 scope.go:117] "RemoveContainer" containerID="f414d4d8e7f690e8237b13fda150cd9bff8d252c2084cf74ba6785e8e70528f4"
Jul 16 17:53:30.034354 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:53:30.034321    2576 scope.go:117] "RemoveContainer" containerID="8c5aed63385b8847ecf45f50c9972309f664536cf0785f89cb4c3ef526673066"
Jul 16 17:54:18.479557 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:18.479201    2576 scope.go:117] "RemoveContainer" containerID="fee343f3206d025617d8169d2f658ac97d6102fde89ee9f66502dfc88b077a59"
Jul 16 17:54:18.786600 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:18.786519    2576 scope.go:117] "RemoveContainer" containerID="013011bdf2cfd9630ab7779e8547b6f948e41fb58d955096a4d4a721e5d29518"
Jul 16 17:54:20.786295 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.786260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerStarted","Data":"e35b4bcc2efe8765be8b26eda7fe87b79accde7e84e7b0bc06b8da945a4baf55"}
Jul 16 17:54:20.786682 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.786315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerStarted","Data":"9bd708cf320d75fc069a102ea359e581bae23c68d40cb9e38fe32a59d9de423c"}
Jul 16 17:54:20.786682 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.786454    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:20.786682 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.786621    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:20.787853 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.787830    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 17:54:20.820094 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:20.820024    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podStartSLOduration=3.400441368 podStartE2EDuration="2m47.820004629s" totalImagesPullingTime="2m15.57903212s" totalInitContainerRuntime="28.840531141s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:51:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:54:20.819621596 +0000 UTC m=+1191.243351656" watchObservedRunningTime="2026-07-16 17:54:20.820004629 +0000 UTC m=+1191.243734691"
Jul 16 17:54:21.790901 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:21.790863    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 17:54:26.795954 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:26.795926    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:26.796501 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:26.796470    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 16 17:54:29.993975 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:29.993945    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:54:29.998023 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:29.997998    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:54:36.796594 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:36.796567    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:44.139659 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.139625    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:54:44.140091 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.139837    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container" containerID="cri-o://9bd708cf320d75fc069a102ea359e581bae23c68d40cb9e38fe32a59d9de423c" gracePeriod=30
Jul 16 17:54:44.140091 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.139888    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kube-rbac-proxy" containerID="cri-o://e35b4bcc2efe8765be8b26eda7fe87b79accde7e84e7b0bc06b8da945a4baf55" gracePeriod=30
Jul 16 17:54:44.883532 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.883490    2576 generic.go:356] "Generic (PLEG): container finished" podID="19284549-063a-45a1-b4bd-aad87f132108" containerID="e35b4bcc2efe8765be8b26eda7fe87b79accde7e84e7b0bc06b8da945a4baf55" exitCode=2
Jul 16 17:54:44.883532 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.883520    2576 generic.go:356] "Generic (PLEG): container finished" podID="19284549-063a-45a1-b4bd-aad87f132108" containerID="9bd708cf320d75fc069a102ea359e581bae23c68d40cb9e38fe32a59d9de423c" exitCode=0
Jul 16 17:54:44.883707 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.883578    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerDied","Data":"e35b4bcc2efe8765be8b26eda7fe87b79accde7e84e7b0bc06b8da945a4baf55"}
Jul 16 17:54:44.883707 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:44.883667    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerDied","Data":"9bd708cf320d75fc069a102ea359e581bae23c68d40cb9e38fe32a59d9de423c"}
Jul 16 17:54:45.016359 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.016333    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:45.132593 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.132562    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls\") pod \"19284549-063a-45a1-b4bd-aad87f132108\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") "
Jul 16 17:54:45.132733 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.132629    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location\") pod \"19284549-063a-45a1-b4bd-aad87f132108\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") "
Jul 16 17:54:45.132733 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.132664    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqssq\" (UniqueName: \"kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq\") pod \"19284549-063a-45a1-b4bd-aad87f132108\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") "
Jul 16 17:54:45.132733 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.132718    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert\") pod \"19284549-063a-45a1-b4bd-aad87f132108\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") "
Jul 16 17:54:45.132921 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.132754    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config\") pod \"19284549-063a-45a1-b4bd-aad87f132108\" (UID: \"19284549-063a-45a1-b4bd-aad87f132108\") "
Jul 16 17:54:45.133091 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.133066    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "19284549-063a-45a1-b4bd-aad87f132108" (UID: "19284549-063a-45a1-b4bd-aad87f132108"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:54:45.133150 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.133118    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "19284549-063a-45a1-b4bd-aad87f132108" (UID: "19284549-063a-45a1-b4bd-aad87f132108"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:54:45.134902 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.134880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq" (OuterVolumeSpecName: "kube-api-access-rqssq") pod "19284549-063a-45a1-b4bd-aad87f132108" (UID: "19284549-063a-45a1-b4bd-aad87f132108"). InnerVolumeSpecName "kube-api-access-rqssq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:54:45.135024 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.135008    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "19284549-063a-45a1-b4bd-aad87f132108" (UID: "19284549-063a-45a1-b4bd-aad87f132108"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:54:45.138365 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.138344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "19284549-063a-45a1-b4bd-aad87f132108" (UID: "19284549-063a-45a1-b4bd-aad87f132108"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:54:45.234172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.234135    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:54:45.234172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.234162    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/19284549-063a-45a1-b4bd-aad87f132108-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:54:45.234172 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.234172    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/19284549-063a-45a1-b4bd-aad87f132108-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:54:45.234604 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.234182    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/19284549-063a-45a1-b4bd-aad87f132108-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:54:45.234604 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.234191    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqssq\" (UniqueName: \"kubernetes.io/projected/19284549-063a-45a1-b4bd-aad87f132108-kube-api-access-rqssq\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:54:45.889630 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.889606    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"
Jul 16 17:54:45.889630 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.889612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq" event={"ID":"19284549-063a-45a1-b4bd-aad87f132108","Type":"ContainerDied","Data":"59883203bfc882a7f3a746cd0049430631cf4ac5cdd127f3aedbbc243462e78f"}
Jul 16 17:54:45.889843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.889678    2576 scope.go:117] "RemoveContainer" containerID="e35b4bcc2efe8765be8b26eda7fe87b79accde7e84e7b0bc06b8da945a4baf55"
Jul 16 17:54:45.898908 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.898884    2576 scope.go:117] "RemoveContainer" containerID="9bd708cf320d75fc069a102ea359e581bae23c68d40cb9e38fe32a59d9de423c"
Jul 16 17:54:45.907843 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.907824    2576 scope.go:117] "RemoveContainer" containerID="092a632d0b3bbebff8e898524672085fba710d9237272195769c71bfe7502bd0"
Jul 16 17:54:45.915857 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.915837    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:54:45.925768 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:45.925750    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-mt8gq"]
Jul 16 17:54:46.090884 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:46.090852    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19284549-063a-45a1-b4bd-aad87f132108" path="/var/lib/kubelet/pods/19284549-063a-45a1-b4bd-aad87f132108/volumes"
Jul 16 17:54:49.476553 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476516    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476790    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476800    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476810    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="storage-initializer"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476816    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="storage-initializer"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476826    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476831    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476840    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476845    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476850    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476855    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476873    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476878    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kube-rbac-proxy"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476885    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="storage-initializer"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476889    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="storage-initializer"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476899    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476904    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476912    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent"
Jul 16 17:54:49.476939 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476918    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476974    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kserve-container"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476982    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="agent"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476988    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kserve-container"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.476994    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kserve-container"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.477001    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="19284549-063a-45a1-b4bd-aad87f132108" containerName="kube-rbac-proxy"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.477006    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b3f2e77-9ed5-4dd7-8aab-87f5373b9bd4" containerName="kube-rbac-proxy"
Jul 16 17:54:49.477479 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.477010    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ba16e4f-7f77-4b1d-a95a-a3280f4230d1" containerName="kube-rbac-proxy"
Jul 16 17:54:49.518504 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.518473    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:54:49.518647 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.518533    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.520878 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.520855    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:54:49.521780 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521756    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:54:49.521780 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521773    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 16 17:54:49.521968 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521802    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 16 17:54:49.521968 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521819    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:54:49.521968 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521759    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-52dqs\""
Jul 16 17:54:49.521968 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.521832    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:54:49.573501 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.573463    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.573649 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.573533    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.573649 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.573562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.573649 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.573587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.573649 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.573606    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgz7x\" (UniqueName: \"kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.674726 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.674688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.674907 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.674731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.674907 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.674857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.674907 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.674893    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgz7x\" (UniqueName: \"kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.675077 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.674960    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.675302 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.675276    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.675532 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.675513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.675616 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.675600    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.677258 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.677240    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.684472 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.684429    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgz7x\" (UniqueName: \"kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x\") pod \"isvc-autogluon-v2-predictor-67b849d456-twczp\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.828350 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.828319    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:49.986858 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:49.985462    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:54:50.911793 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:50.911757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerStarted","Data":"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"}
Jul 16 17:54:50.912154 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:50.911819    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerStarted","Data":"c4b41ec713371676c4cb9e67bcdbf19c5b8e5e964981ae8160d760d6580830f5"}
Jul 16 17:54:55.933948 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:55.933912    2576 generic.go:356] "Generic (PLEG): container finished" podID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerID="9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef" exitCode=0
Jul 16 17:54:55.934353 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:55.933973    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerDied","Data":"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"}
Jul 16 17:54:56.940090 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:56.940056    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerStarted","Data":"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"}
Jul 16 17:54:56.940512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:56.940121    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerStarted","Data":"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"}
Jul 16 17:54:56.940512 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:56.940325    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:54:56.964736 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:56.964677    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" podStartSLOduration=3.172586966 podStartE2EDuration="7.964659835s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.792072869s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:54:56.963794646 +0000 UTC m=+1227.387524710" watchObservedRunningTime="2026-07-16 17:54:56.964659835 +0000 UTC m=+1227.388389895"
Jul 16 17:54:57.945088 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:54:57.945046    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:55:03.954833 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:55:03.954800    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:56:27.089577 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:27.089535    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:56:29.637295 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:29.637265    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:56:29.637703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:29.637494    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kserve-container" containerID="cri-o://b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123" gracePeriod=30
Jul 16 17:56:29.637703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:29.637576    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kube-rbac-proxy" containerID="cri-o://a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1" gracePeriod=30
Jul 16 17:56:30.292878 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.292846    2576 generic.go:356] "Generic (PLEG): container finished" podID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerID="a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1" exitCode=2
Jul 16 17:56:30.293014 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.292914    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerDied","Data":"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"}
Jul 16 17:56:30.477267 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.477241    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:56:30.501177 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501152    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config\") pod \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") "
Jul 16 17:56:30.501288 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert\") pod \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") "
Jul 16 17:56:30.501288 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501230    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls\") pod \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") "
Jul 16 17:56:30.501288 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501249    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location\") pod \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") "
Jul 16 17:56:30.501431 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501325    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgz7x\" (UniqueName: \"kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x\") pod \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\" (UID: \"c7f9c7c9-a231-4303-a9ed-59597d199cd3\") "
Jul 16 17:56:30.501698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501648    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c7f9c7c9-a231-4303-a9ed-59597d199cd3" (UID: "c7f9c7c9-a231-4303-a9ed-59597d199cd3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:30.501698 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.501568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c7f9c7c9-a231-4303-a9ed-59597d199cd3" (UID: "c7f9c7c9-a231-4303-a9ed-59597d199cd3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:56:30.503637 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.503609    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x" (OuterVolumeSpecName: "kube-api-access-wgz7x") pod "c7f9c7c9-a231-4303-a9ed-59597d199cd3" (UID: "c7f9c7c9-a231-4303-a9ed-59597d199cd3"). InnerVolumeSpecName "kube-api-access-wgz7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:56:30.503856 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.503833    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c7f9c7c9-a231-4303-a9ed-59597d199cd3" (UID: "c7f9c7c9-a231-4303-a9ed-59597d199cd3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:56:30.507884 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.507858    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c7f9c7c9-a231-4303-a9ed-59597d199cd3" (UID: "c7f9c7c9-a231-4303-a9ed-59597d199cd3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:56:30.602703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.602670    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:56:30.602703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.602698    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c7f9c7c9-a231-4303-a9ed-59597d199cd3-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:56:30.602703 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.602708    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c7f9c7c9-a231-4303-a9ed-59597d199cd3-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:56:30.602931 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.602715    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:56:30.602931 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:30.602724    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgz7x\" (UniqueName: \"kubernetes.io/projected/c7f9c7c9-a231-4303-a9ed-59597d199cd3-kube-api-access-wgz7x\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:56:31.298913 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.298874    2576 generic.go:356] "Generic (PLEG): container finished" podID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerID="b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123" exitCode=0
Jul 16 17:56:31.299329 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.298930    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"
Jul 16 17:56:31.299329 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.298959    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerDied","Data":"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"}
Jul 16 17:56:31.299329 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.299019    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp" event={"ID":"c7f9c7c9-a231-4303-a9ed-59597d199cd3","Type":"ContainerDied","Data":"c4b41ec713371676c4cb9e67bcdbf19c5b8e5e964981ae8160d760d6580830f5"}
Jul 16 17:56:31.299329 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.299046    2576 scope.go:117] "RemoveContainer" containerID="a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"
Jul 16 17:56:31.307950 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.307932    2576 scope.go:117] "RemoveContainer" containerID="b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"
Jul 16 17:56:31.315130 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.315114    2576 scope.go:117] "RemoveContainer" containerID="9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"
Jul 16 17:56:31.321993 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.321952    2576 scope.go:117] "RemoveContainer" containerID="a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"
Jul 16 17:56:31.322391 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:56:31.322360    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1\": container with ID starting with a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1 not found: ID does not exist" containerID="a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"
Jul 16 17:56:31.322668 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.322402    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1"} err="failed to get container status \"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1\": rpc error: code = NotFound desc = could not find container \"a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1\": container with ID starting with a77da60491336e6cf4a51c774ed5e62782fc883e7721546f51447b0185e423f1 not found: ID does not exist"
Jul 16 17:56:31.322668 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.322427    2576 scope.go:117] "RemoveContainer" containerID="b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"
Jul 16 17:56:31.322856 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:56:31.322833    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123\": container with ID starting with b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123 not found: ID does not exist" containerID="b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"
Jul 16 17:56:31.322897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.322864    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123"} err="failed to get container status \"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123\": rpc error: code = NotFound desc = could not find container \"b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123\": container with ID starting with b24c117dc1e9978e9732de84bcf98e36ba521cc40041bc188cf220adbfe87123 not found: ID does not exist"
Jul 16 17:56:31.322897 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.322880    2576 scope.go:117] "RemoveContainer" containerID="9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"
Jul 16 17:56:31.323150 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:56:31.323126    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef\": container with ID starting with 9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef not found: ID does not exist" containerID="9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"
Jul 16 17:56:31.323206 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.323186    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef"} err="failed to get container status \"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef\": rpc error: code = NotFound desc = could not find container \"9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef\": container with ID starting with 9e4571e8d633eb15f0bd73fd294f2b92b3a884230af337732e45900bf2d431ef not found: ID does not exist"
Jul 16 17:56:31.323999 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.323982    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:56:31.331205 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:31.331183    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-twczp"]
Jul 16 17:56:32.091211 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:32.091180    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" path="/var/lib/kubelet/pods/c7f9c7c9-a231-4303-a9ed-59597d199cd3/volumes"
Jul 16 17:56:34.765228 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765197    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765639    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="storage-initializer"
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765658    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="storage-initializer"
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765669    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kube-rbac-proxy"
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765677    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kube-rbac-proxy"
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765694    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kserve-container"
Jul 16 17:56:34.765734 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765703    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kserve-container"
Jul 16 17:56:34.766090 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765783    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kserve-container"
Jul 16 17:56:34.766090 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.765794    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7f9c7c9-a231-4303-a9ed-59597d199cd3" containerName="kube-rbac-proxy"
Jul 16 17:56:34.769034 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.769011    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.771648 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.771623    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 16 17:56:34.771747 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.771638    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:56:34.771747 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.771654    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 16 17:56:34.771747 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.771727    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-52dqs\""
Jul 16 17:56:34.772016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.771998    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:56:34.772589 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.772573    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:56:34.772655 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.772608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:56:34.780838 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.780817    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:56:34.833785 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.833752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mdf\" (UniqueName: \"kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.833947 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.833808    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.833947 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.833907    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.834017 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.833959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.834017 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.834007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.934792 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.934755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.934792 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.934797    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935003 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.934818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935003 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:56:34.934906    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 16 17:56:34.935003 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.934940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mdf\" (UniqueName: \"kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935003 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:56:34.934961    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls podName:7ebefffc-b21f-40bd-8f6e-f6aea8a62f04 nodeName:}" failed. No retries permitted until 2026-07-16 17:56:35.434947004 +0000 UTC m=+1325.858677043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 16 17:56:34.935154 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.935019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935188 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.935159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935504 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.935485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.935622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.935606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:34.944685 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:34.944665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mdf\" (UniqueName: \"kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:35.437678 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:35.437642    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:35.440065 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:35.440029    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:35.679258 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:35.679220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:35.809307 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:35.809279    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:56:35.811820 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:56:35.811792    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ebefffc_b21f_40bd_8f6e_f6aea8a62f04.slice/crio-c42dd1a361bca6a46edd9175e0e2223eddc16ec2c3d1598ba67c7593696472f4 WatchSource:0}: Error finding container c42dd1a361bca6a46edd9175e0e2223eddc16ec2c3d1598ba67c7593696472f4: Status 404 returned error can't find the container with id c42dd1a361bca6a46edd9175e0e2223eddc16ec2c3d1598ba67c7593696472f4
Jul 16 17:56:36.326604 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:36.326567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerStarted","Data":"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"}
Jul 16 17:56:36.326801 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:36.326631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerStarted","Data":"c42dd1a361bca6a46edd9175e0e2223eddc16ec2c3d1598ba67c7593696472f4"}
Jul 16 17:56:57.405498 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:57.405456    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerID="8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666" exitCode=0
Jul 16 17:56:57.405887 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:57.405470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerDied","Data":"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"}
Jul 16 17:56:58.412076 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:58.412039    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerStarted","Data":"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"}
Jul 16 17:56:58.412684 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:58.412102    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerStarted","Data":"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"}
Jul 16 17:56:58.412684 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:58.412298    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:56:58.432344 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:58.432290    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" podStartSLOduration=3.8280469679999998 podStartE2EDuration="24.432270788s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.60422382s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:56:58.430181378 +0000 UTC m=+1348.853911438" watchObservedRunningTime="2026-07-16 17:56:58.432270788 +0000 UTC m=+1348.856000848"
Jul 16 17:56:59.416481 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:56:59.416421    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:57:05.426666 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:57:05.426639    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:58:32.092150 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:32.092077    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:58:34.921505 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:34.921461    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:58:34.921970 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:34.921751    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kserve-container" containerID="cri-o://96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9" gracePeriod=30
Jul 16 17:58:34.921970 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:34.921814    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kube-rbac-proxy" containerID="cri-o://e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92" gracePeriod=30
Jul 16 17:58:35.421837 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.421793    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 16 17:58:35.770027 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.770000    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:58:35.777125 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777102    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerID="e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92" exitCode=2
Jul 16 17:58:35.777125 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777122    2576 generic.go:356] "Generic (PLEG): container finished" podID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerID="96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9" exitCode=0
Jul 16 17:58:35.777264 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777151    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"
Jul 16 17:58:35.777264 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777179    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerDied","Data":"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"}
Jul 16 17:58:35.777264 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerDied","Data":"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"}
Jul 16 17:58:35.777264 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm" event={"ID":"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04","Type":"ContainerDied","Data":"c42dd1a361bca6a46edd9175e0e2223eddc16ec2c3d1598ba67c7593696472f4"}
Jul 16 17:58:35.777386 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.777284    2576 scope.go:117] "RemoveContainer" containerID="e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"
Jul 16 17:58:35.785008 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.784993    2576 scope.go:117] "RemoveContainer" containerID="96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"
Jul 16 17:58:35.794298 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.793484    2576 scope.go:117] "RemoveContainer" containerID="8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"
Jul 16 17:58:35.802700 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.802675    2576 scope.go:117] "RemoveContainer" containerID="e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"
Jul 16 17:58:35.802995 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:58:35.802973    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92\": container with ID starting with e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92 not found: ID does not exist" containerID="e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"
Jul 16 17:58:35.803068 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803002    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"} err="failed to get container status \"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92\": rpc error: code = NotFound desc = could not find container \"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92\": container with ID starting with e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92 not found: ID does not exist"
Jul 16 17:58:35.803068 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803017    2576 scope.go:117] "RemoveContainer" containerID="96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"
Jul 16 17:58:35.803293 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:58:35.803275    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9\": container with ID starting with 96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9 not found: ID does not exist" containerID="96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"
Jul 16 17:58:35.803334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803312    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"} err="failed to get container status \"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9\": rpc error: code = NotFound desc = could not find container \"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9\": container with ID starting with 96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9 not found: ID does not exist"
Jul 16 17:58:35.803334 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803331    2576 scope.go:117] "RemoveContainer" containerID="8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"
Jul 16 17:58:35.803608 ip-10-0-143-77 kubenswrapper[2576]: E0716 17:58:35.803593    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666\": container with ID starting with 8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666 not found: ID does not exist" containerID="8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"
Jul 16 17:58:35.803672 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803610    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"} err="failed to get container status \"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666\": rpc error: code = NotFound desc = could not find container \"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666\": container with ID starting with 8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666 not found: ID does not exist"
Jul 16 17:58:35.803672 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803628    2576 scope.go:117] "RemoveContainer" containerID="e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"
Jul 16 17:58:35.803855 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803836    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92"} err="failed to get container status \"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92\": rpc error: code = NotFound desc = could not find container \"e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92\": container with ID starting with e233243e092196689cf1981fc92f81dac36ecdc88c5e4b08b5659fd6eb33fc92 not found: ID does not exist"
Jul 16 17:58:35.803894 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.803856    2576 scope.go:117] "RemoveContainer" containerID="96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"
Jul 16 17:58:35.804081 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.804066    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9"} err="failed to get container status \"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9\": rpc error: code = NotFound desc = could not find container \"96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9\": container with ID starting with 96fe1f51aed0fc3f99eb1f18ae027a27e06a5b612a3589965174993e6a8721f9 not found: ID does not exist"
Jul 16 17:58:35.804130 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.804082    2576 scope.go:117] "RemoveContainer" containerID="8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"
Jul 16 17:58:35.804320 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.804296    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666"} err="failed to get container status \"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666\": rpc error: code = NotFound desc = could not find container \"8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666\": container with ID starting with 8793db7312c25e0913a091750e741bac9d889b9a7778008145c9e010a9124666 not found: ID does not exist"
Jul 16 17:58:35.825623 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.825600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location\") pod \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") "
Jul 16 17:58:35.825721 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.825632    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert\") pod \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") "
Jul 16 17:58:35.825721 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.825668    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config\") pod \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") "
Jul 16 17:58:35.825721 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.825712    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") pod \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") "
Jul 16 17:58:35.825888 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.825737    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8mdf\" (UniqueName: \"kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf\") pod \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\" (UID: \"7ebefffc-b21f-40bd-8f6e-f6aea8a62f04\") "
Jul 16 17:58:35.826102 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.826081    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:58:35.826189 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.826161    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 17:58:35.827983 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.827963    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf" (OuterVolumeSpecName: "kube-api-access-m8mdf") pod "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04"). InnerVolumeSpecName "kube-api-access-m8mdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 17:58:35.828089 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.827977    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 17:58:35.830483 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.830420    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" (UID: "7ebefffc-b21f-40bd-8f6e-f6aea8a62f04"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 17:58:35.926412 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.926384    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8mdf\" (UniqueName: \"kubernetes.io/projected/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-api-access-m8mdf\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:58:35.926412 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.926405    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:58:35.926790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.926417    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:58:35.926790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.926430    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:58:35.926790 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:35.926459    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 17:58:36.099953 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:36.099921    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:58:36.104558 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:36.104534    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-cf5rm"]
Jul 16 17:58:38.090924 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:38.090884    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" path="/var/lib/kubelet/pods/7ebefffc-b21f-40bd-8f6e-f6aea8a62f04/volumes"
Jul 16 17:58:40.104590 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104562    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104834    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="storage-initializer"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104845    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="storage-initializer"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104852    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kserve-container"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104858    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kserve-container"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104867    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kube-rbac-proxy"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104872    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kube-rbac-proxy"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104928    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kube-rbac-proxy"
Jul 16 17:58:40.104951 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.104937    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ebefffc-b21f-40bd-8f6e-f6aea8a62f04" containerName="kserve-container"
Jul 16 17:58:40.109599 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.109582    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.111959 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.111937    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 16 17:58:40.112201 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.112182    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 17:58:40.112289 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.112260    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 16 17:58:40.112339 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.112318    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 17:58:40.112894 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.112877    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 17:58:40.113062 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.113050    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-52dqs\""
Jul 16 17:58:40.113115 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.113076    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 17:58:40.119660 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.119637    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 17:58:40.162351 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.162319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.162351 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.162354    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.162568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.162385    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blb2g\" (UniqueName: \"kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.162568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.162502    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.162568 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.162526    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.263712 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.263677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.263712 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.263710    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.263938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.263749    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.263938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.263769    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.263938 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.263789    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blb2g\" (UniqueName: \"kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.264284 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.264260    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.264510 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.264451    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.264510 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.264493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.266230 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.266204    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.273016 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.272994    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blb2g\" (UniqueName: \"kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.420356 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.420273    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:40.556553 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.556519    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 17:58:40.559582 ip-10-0-143-77 kubenswrapper[2576]: W0716 17:58:40.559553    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32e2ae6_1d55_45b5_9dd7_4adcc5d5f4da.slice/crio-a7cc78e213b3d836f2084af2c3c7912283be70454bde7e274b902b75caa3178e WatchSource:0}: Error finding container a7cc78e213b3d836f2084af2c3c7912283be70454bde7e274b902b75caa3178e: Status 404 returned error can't find the container with id a7cc78e213b3d836f2084af2c3c7912283be70454bde7e274b902b75caa3178e
Jul 16 17:58:40.801270 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.801186    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerStarted","Data":"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"}
Jul 16 17:58:40.801270 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:40.801248    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerStarted","Data":"a7cc78e213b3d836f2084af2c3c7912283be70454bde7e274b902b75caa3178e"}
Jul 16 17:58:45.824770 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:45.824737    2576 generic.go:356] "Generic (PLEG): container finished" podID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerID="89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1" exitCode=0
Jul 16 17:58:45.825116 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:45.824812    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerDied","Data":"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"}
Jul 16 17:58:46.831457 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:46.831406    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerStarted","Data":"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"}
Jul 16 17:58:46.831845 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:46.831492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerStarted","Data":"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"}
Jul 16 17:58:46.831845 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:46.831711    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:46.831932 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:46.831867    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:58:46.851391 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:46.851336    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" podStartSLOduration=1.763666307 podStartE2EDuration="6.851321197s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.08765489s" isStatefulPod=false podCreationTimestamp="2026-07-16 17:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 17:58:46.849768951 +0000 UTC m=+1457.273499012" watchObservedRunningTime="2026-07-16 17:58:46.851321197 +0000 UTC m=+1457.275051322"
Jul 16 17:58:52.841055 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:58:52.841025    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 17:59:30.016622 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:59:30.016597    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 17:59:30.025057 ip-10-0-143-77 kubenswrapper[2576]: I0716 17:59:30.025035    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 18:00:15.090118 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:15.090028    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 18:00:20.250751 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:20.250720    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 18:00:20.253086 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:20.250960    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kserve-container" containerID="cri-o://f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af" gracePeriod=30
Jul 16 18:00:20.253086 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:20.250985    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kube-rbac-proxy" containerID="cri-o://c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e" gracePeriod=30
Jul 16 18:00:21.099809 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.099784    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 18:00:21.178018 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.177938    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls\") pod \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") "
Jul 16 18:00:21.178018 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.177991    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert\") pod \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") "
Jul 16 18:00:21.178018 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.178010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blb2g\" (UniqueName: \"kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g\") pod \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") "
Jul 16 18:00:21.178287 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.178031    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config\") pod \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") "
Jul 16 18:00:21.178287 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.178077    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location\") pod \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\" (UID: \"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da\") "
Jul 16 18:00:21.178484 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.178430    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" (UID: "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:21.178484 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.178471    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" (UID: "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:21.180897 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.180871    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" (UID: "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:00:21.180897 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.180880    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g" (OuterVolumeSpecName: "kube-api-access-blb2g") pod "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" (UID: "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da"). InnerVolumeSpecName "kube-api-access-blb2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:00:21.182312 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.182289    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" (UID: "b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:00:21.188377 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188353    2576 generic.go:356] "Generic (PLEG): container finished" podID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerID="c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e" exitCode=2
Jul 16 18:00:21.188377 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188375    2576 generic.go:356] "Generic (PLEG): container finished" podID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerID="f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af" exitCode=0
Jul 16 18:00:21.188537 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"
Jul 16 18:00:21.188537 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188429    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerDied","Data":"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"}
Jul 16 18:00:21.188537 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188502    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerDied","Data":"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"}
Jul 16 18:00:21.188537 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188523    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k" event={"ID":"b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da","Type":"ContainerDied","Data":"a7cc78e213b3d836f2084af2c3c7912283be70454bde7e274b902b75caa3178e"}
Jul 16 18:00:21.188694 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.188549    2576 scope.go:117] "RemoveContainer" containerID="c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"
Jul 16 18:00:21.196644 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.196625    2576 scope.go:117] "RemoveContainer" containerID="f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"
Jul 16 18:00:21.204213 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.204197    2576 scope.go:117] "RemoveContainer" containerID="89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"
Jul 16 18:00:21.211098 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211078    2576 scope.go:117] "RemoveContainer" containerID="c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"
Jul 16 18:00:21.211343 ip-10-0-143-77 kubenswrapper[2576]: E0716 18:00:21.211323    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e\": container with ID starting with c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e not found: ID does not exist" containerID="c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"
Jul 16 18:00:21.211392 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211351    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"} err="failed to get container status \"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e\": rpc error: code = NotFound desc = could not find container \"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e\": container with ID starting with c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e not found: ID does not exist"
Jul 16 18:00:21.211392 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211369    2576 scope.go:117] "RemoveContainer" containerID="f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"
Jul 16 18:00:21.211633 ip-10-0-143-77 kubenswrapper[2576]: E0716 18:00:21.211616    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af\": container with ID starting with f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af not found: ID does not exist" containerID="f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"
Jul 16 18:00:21.211676 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211639    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"} err="failed to get container status \"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af\": rpc error: code = NotFound desc = could not find container \"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af\": container with ID starting with f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af not found: ID does not exist"
Jul 16 18:00:21.211676 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211654    2576 scope.go:117] "RemoveContainer" containerID="89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"
Jul 16 18:00:21.211916 ip-10-0-143-77 kubenswrapper[2576]: E0716 18:00:21.211860    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1\": container with ID starting with 89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1 not found: ID does not exist" containerID="89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"
Jul 16 18:00:21.212014 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211925    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"} err="failed to get container status \"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1\": rpc error: code = NotFound desc = could not find container \"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1\": container with ID starting with 89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1 not found: ID does not exist"
Jul 16 18:00:21.212014 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.211939    2576 scope.go:117] "RemoveContainer" containerID="c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"
Jul 16 18:00:21.212318 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.212285    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e"} err="failed to get container status \"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e\": rpc error: code = NotFound desc = could not find container \"c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e\": container with ID starting with c885f214df91c4c95e416587a82847d7450452db74bfc2ef658b92e8e655e92e not found: ID does not exist"
Jul 16 18:00:21.212318 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.212312    2576 scope.go:117] "RemoveContainer" containerID="f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"
Jul 16 18:00:21.212940 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.212919    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af"} err="failed to get container status \"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af\": rpc error: code = NotFound desc = could not find container \"f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af\": container with ID starting with f5af8a836385ba58820f884605a85f54b94e29a5634c07e1f5c3f5822bda47af not found: ID does not exist"
Jul 16 18:00:21.213012 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.212942    2576 scope.go:117] "RemoveContainer" containerID="89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"
Jul 16 18:00:21.213213 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.213194    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1"} err="failed to get container status \"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1\": rpc error: code = NotFound desc = could not find container \"89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1\": container with ID starting with 89d347543073256489cf916f1a95d073eb08796b651ea52477cc7d11252febd1 not found: ID does not exist"
Jul 16 18:00:21.214213 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.214195    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 18:00:21.218965 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.218947    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-jmw9k"]
Jul 16 18:00:21.278871 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.278841    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:21.278871 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.278865    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:21.279231 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.278875    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:21.279231 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.278896    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blb2g\" (UniqueName: \"kubernetes.io/projected/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-api-access-blb2g\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:21.279231 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:21.278907    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:22.090602 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:22.090570    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" path="/var/lib/kubelet/pods/b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da/volumes"
Jul 16 18:00:25.497999 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.497922    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498189    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kserve-container"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498199    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kserve-container"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498218    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kube-rbac-proxy"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498224    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kube-rbac-proxy"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498234    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="storage-initializer"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498240    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="storage-initializer"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498286    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kube-rbac-proxy"
Jul 16 18:00:25.498328 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.498293    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32e2ae6-1d55-45b5-9dd7-4adcc5d5f4da" containerName="kserve-container"
Jul 16 18:00:25.502954 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.502933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.505795 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.505772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 18:00:25.505939 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.505828    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 16 18:00:25.505939 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.505777    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 16 18:00:25.505939 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.505771    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 16 18:00:25.506074 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.505946    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-52dqs\""
Jul 16 18:00:25.506672 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.506655    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 16 18:00:25.506785 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.506772    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 18:00:25.512401 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.512376    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:25.610571 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.610539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.610571 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.610579    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwcn7\" (UniqueName: \"kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.610783 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.610665    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.610783 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.610724    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.610783 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.610758    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.711684 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.711647    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.711821 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.711703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.711914 ip-10-0-143-77 kubenswrapper[2576]: E0716 18:00:25.711811    2576 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-serving-cert: secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 16 18:00:25.711914 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.711811    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.711914 ip-10-0-143-77 kubenswrapper[2576]: E0716 18:00:25.711880    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls podName:13f6cc07-57f0-468b-bd8e-6e349e2d8024 nodeName:}" failed. No retries permitted until 2026-07-16 18:00:26.211860414 +0000 UTC m=+1556.635590454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls") pod "isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024") : secret "isvc-autogluon-ts-v1-predictor-serving-cert" not found
Jul 16 18:00:25.712075 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.711927    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.712075 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.711971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwcn7\" (UniqueName: \"kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.712181 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.712109    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.712321 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.712301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.712538 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.712518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:25.721114 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:25.721090    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwcn7\" (UniqueName: \"kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:26.215999 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:26.215964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:26.218472 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:26.218428    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:26.414209 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:26.414170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:26.549862 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:26.549837    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:26.552373 ip-10-0-143-77 kubenswrapper[2576]: W0716 18:00:26.552345    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f6cc07_57f0_468b_bd8e_6e349e2d8024.slice/crio-3ecb8fc0c2df873bf93529e59192f5ac81756cc53f0e2e09ef00c5c134e6411e WatchSource:0}: Error finding container 3ecb8fc0c2df873bf93529e59192f5ac81756cc53f0e2e09ef00c5c134e6411e: Status 404 returned error can't find the container with id 3ecb8fc0c2df873bf93529e59192f5ac81756cc53f0e2e09ef00c5c134e6411e
Jul 16 18:00:27.219909 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:27.219870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerStarted","Data":"d79b90099fa238d9f7b92215f51d9d88ff8bf174665d294a57ac71203f1d3d55"}
Jul 16 18:00:27.220106 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:27.219954    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerStarted","Data":"3ecb8fc0c2df873bf93529e59192f5ac81756cc53f0e2e09ef00c5c134e6411e"}
Jul 16 18:00:35.255929 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:35.255894    2576 generic.go:356] "Generic (PLEG): container finished" podID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerID="d79b90099fa238d9f7b92215f51d9d88ff8bf174665d294a57ac71203f1d3d55" exitCode=0
Jul 16 18:00:35.256302 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:35.255970    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerDied","Data":"d79b90099fa238d9f7b92215f51d9d88ff8bf174665d294a57ac71203f1d3d55"}
Jul 16 18:00:36.262777 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:36.262746    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerStarted","Data":"4b41f4e8b64f4264cc66b8a49cc72307d81c35d4d1c95d7fa5f8dffb6cf9432e"}
Jul 16 18:00:36.263179 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:36.262813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerStarted","Data":"04c0b636f54b4bd3aa61d2f892bd97362e077275bf78356af1b7153ccb05b893"}
Jul 16 18:00:36.263179 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:36.263134    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:36.283057 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:36.283003    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podStartSLOduration=3.549212185 podStartE2EDuration="11.28298731s" totalImagesPullingTime="0s" totalInitContainerRuntime="7.733775125s" isStatefulPod=false podCreationTimestamp="2026-07-16 18:00:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:00:36.282188266 +0000 UTC m=+1566.705918327" watchObservedRunningTime="2026-07-16 18:00:36.28298731 +0000 UTC m=+1566.706717369"
Jul 16 18:00:37.267632 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:37.267593    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:37.268957 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:37.268925    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 16 18:00:38.271619 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:38.271574    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 16 18:00:43.276397 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:43.276362    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:43.276957 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:43.276933    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 16 18:00:53.278130 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:53.278098    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:55.522505 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:55.522475    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:55.522902 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:55.522683    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container" containerID="cri-o://04c0b636f54b4bd3aa61d2f892bd97362e077275bf78356af1b7153ccb05b893" gracePeriod=30
Jul 16 18:00:55.522902 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:55.522736    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kube-rbac-proxy" containerID="cri-o://4b41f4e8b64f4264cc66b8a49cc72307d81c35d4d1c95d7fa5f8dffb6cf9432e" gracePeriod=30
Jul 16 18:00:56.348715 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.348666    2576 generic.go:356] "Generic (PLEG): container finished" podID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerID="4b41f4e8b64f4264cc66b8a49cc72307d81c35d4d1c95d7fa5f8dffb6cf9432e" exitCode=2
Jul 16 18:00:56.348715 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.348700    2576 generic.go:356] "Generic (PLEG): container finished" podID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerID="04c0b636f54b4bd3aa61d2f892bd97362e077275bf78356af1b7153ccb05b893" exitCode=0
Jul 16 18:00:56.348892 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.348735    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerDied","Data":"4b41f4e8b64f4264cc66b8a49cc72307d81c35d4d1c95d7fa5f8dffb6cf9432e"}
Jul 16 18:00:56.348892 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.348792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerDied","Data":"04c0b636f54b4bd3aa61d2f892bd97362e077275bf78356af1b7153ccb05b893"}
Jul 16 18:00:56.371163 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.371140    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:56.471647 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.471566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") pod \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") "
Jul 16 18:00:56.471647 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.471618    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config\") pod \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") "
Jul 16 18:00:56.471873 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.471653    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwcn7\" (UniqueName: \"kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7\") pod \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") "
Jul 16 18:00:56.471873 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.471750    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert\") pod \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") "
Jul 16 18:00:56.471873 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.471828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location\") pod \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\" (UID: \"13f6cc07-57f0-468b-bd8e-6e349e2d8024\") "
Jul 16 18:00:56.472090 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.472067    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "13f6cc07-57f0-468b-bd8e-6e349e2d8024" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:56.472184 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.472162    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "13f6cc07-57f0-468b-bd8e-6e349e2d8024" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 18:00:56.472669 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.472650    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "13f6cc07-57f0-468b-bd8e-6e349e2d8024" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 18:00:56.473898 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.473876    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "13f6cc07-57f0-468b-bd8e-6e349e2d8024" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 18:00:56.473965 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.473921    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7" (OuterVolumeSpecName: "kube-api-access-jwcn7") pod "13f6cc07-57f0-468b-bd8e-6e349e2d8024" (UID: "13f6cc07-57f0-468b-bd8e-6e349e2d8024"). InnerVolumeSpecName "kube-api-access-jwcn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 18:00:56.572816 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.572769    2576 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kserve-provision-location\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:56.572816 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.572808    2576 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/13f6cc07-57f0-468b-bd8e-6e349e2d8024-proxy-tls\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:56.572816 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.572821    2576 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-rbac-proxy-sar-config\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:56.572816 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.572832    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwcn7\" (UniqueName: \"kubernetes.io/projected/13f6cc07-57f0-468b-bd8e-6e349e2d8024-kube-api-access-jwcn7\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:56.573278 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:56.572841    2576 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/13f6cc07-57f0-468b-bd8e-6e349e2d8024-cabundle-cert\") on node \"ip-10-0-143-77.ec2.internal\" DevicePath \"\""
Jul 16 18:00:57.354539 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.354505    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2" event={"ID":"13f6cc07-57f0-468b-bd8e-6e349e2d8024","Type":"ContainerDied","Data":"3ecb8fc0c2df873bf93529e59192f5ac81756cc53f0e2e09ef00c5c134e6411e"}
Jul 16 18:00:57.354539 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.354537    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"
Jul 16 18:00:57.354787 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.354582    2576 scope.go:117] "RemoveContainer" containerID="4b41f4e8b64f4264cc66b8a49cc72307d81c35d4d1c95d7fa5f8dffb6cf9432e"
Jul 16 18:00:57.362912 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.362893    2576 scope.go:117] "RemoveContainer" containerID="04c0b636f54b4bd3aa61d2f892bd97362e077275bf78356af1b7153ccb05b893"
Jul 16 18:00:57.370257 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.370239    2576 scope.go:117] "RemoveContainer" containerID="d79b90099fa238d9f7b92215f51d9d88ff8bf174665d294a57ac71203f1d3d55"
Jul 16 18:00:57.377394 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.377369    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:57.381723 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:57.381701    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-2c2f2"]
Jul 16 18:00:58.090717 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:00:58.090679    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" path="/var/lib/kubelet/pods/13f6cc07-57f0-468b-bd8e-6e349e2d8024/volumes"
Jul 16 18:01:34.920744 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:34.920715    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4z97z_854d98e3-464a-4550-b809-e0964d0c8175/global-pull-secret-syncer/0.log"
Jul 16 18:01:35.107359 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:35.107334    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-n2sbn_0d83f116-f57c-42b7-aa3e-06d562f2cc05/konnectivity-agent/0.log"
Jul 16 18:01:35.221389 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:35.221310    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-77.ec2.internal_648d6c1142d0e1d3e99bb10c691cfa84/haproxy/0.log"
Jul 16 18:01:39.727056 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:39.727030    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-55d48d7796-bvgzx_34dd1bf1-45c9-4a2b-a74d-7ea2b49243bc/metrics-server/0.log"
Jul 16 18:01:39.764000 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:39.763977    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5f68df7f58-tt5b7_b5a3ce9b-4e6b-4393-9198-ba4ee1c66ff8/monitoring-plugin/0.log"
Jul 16 18:01:39.983702 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:39.983628    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8mr2_6e883712-9c0c-452e-a959-380785443207/node-exporter/0.log"
Jul 16 18:01:40.015342 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:40.015317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8mr2_6e883712-9c0c-452e-a959-380785443207/kube-rbac-proxy/0.log"
Jul 16 18:01:40.048551 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:40.048526    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-t8mr2_6e883712-9c0c-452e-a959-380785443207/init-textfile/0.log"
Jul 16 18:01:40.413996 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:40.413966    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-cc6sv_516ab859-9ccd-4464-bdf8-9b367616d872/prometheus-operator/0.log"
Jul 16 18:01:40.439200 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:40.439170    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-cc6sv_516ab859-9ccd-4464-bdf8-9b367616d872/kube-rbac-proxy/0.log"
Jul 16 18:01:43.683921 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.683891    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"]
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684193    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kube-rbac-proxy"
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684203    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kube-rbac-proxy"
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684212    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="storage-initializer"
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684219    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="storage-initializer"
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684229    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container"
Jul 16 18:01:43.684269 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684234    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container"
Jul 16 18:01:43.684471 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684284    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kube-rbac-proxy"
Jul 16 18:01:43.684471 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.684293    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="13f6cc07-57f0-468b-bd8e-6e349e2d8024" containerName="kserve-container"
Jul 16 18:01:43.687183 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.687165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.689649 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.689625    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x48rf\"/\"kube-root-ca.crt\""
Jul 16 18:01:43.690626 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.690607    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-x48rf\"/\"default-dockercfg-zzqfb\""
Jul 16 18:01:43.690747 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.690625    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-x48rf\"/\"openshift-service-ca.crt\""
Jul 16 18:01:43.695481 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.695458    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"]
Jul 16 18:01:43.830234 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.830209    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-zck5v_dbd576ff-bc74-4b42-ad1a-1610928fb9f3/volume-data-source-validator/0.log"
Jul 16 18:01:43.848946 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.848912    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-lib-modules\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.849109 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.848965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-sys\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.849109 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.849036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9tmt\" (UniqueName: \"kubernetes.io/projected/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-kube-api-access-l9tmt\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.849109 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.849076    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-proc\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.849281 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.849113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-podres\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949564 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949470    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-lib-modules\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949564 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949524    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-sys\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949564 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9tmt\" (UniqueName: \"kubernetes.io/projected/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-kube-api-access-l9tmt\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949580    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-proc\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949611    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-podres\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-sys\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-lib-modules\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949702    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-proc\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.949801 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.949721    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-podres\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.958633 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.958611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9tmt\" (UniqueName: \"kubernetes.io/projected/8849a96d-dc2a-4078-ba3e-9f9f61e547fb-kube-api-access-l9tmt\") pod \"perf-node-gather-daemonset-5vj2k\" (UID: \"8849a96d-dc2a-4078-ba3e-9f9f61e547fb\") " pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:43.997314 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:43.997290    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:44.123820 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.123797    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"]
Jul 16 18:01:44.126416 ip-10-0-143-77 kubenswrapper[2576]: W0716 18:01:44.126387    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8849a96d_dc2a_4078_ba3e_9f9f61e547fb.slice/crio-d3a6b71e409f95db23cbc45ebc8a574213253af8908ec14c1ffb0dbc1de5e463 WatchSource:0}: Error finding container d3a6b71e409f95db23cbc45ebc8a574213253af8908ec14c1ffb0dbc1de5e463: Status 404 returned error can't find the container with id d3a6b71e409f95db23cbc45ebc8a574213253af8908ec14c1ffb0dbc1de5e463
Jul 16 18:01:44.530747 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.530710    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k" event={"ID":"8849a96d-dc2a-4078-ba3e-9f9f61e547fb","Type":"ContainerStarted","Data":"b1608482e48b5a38e4775f3c1ba4f35e565b70ed2f1e0373ebb0c6696d3e4088"}
Jul 16 18:01:44.530921 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.530765    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k" event={"ID":"8849a96d-dc2a-4078-ba3e-9f9f61e547fb","Type":"ContainerStarted","Data":"d3a6b71e409f95db23cbc45ebc8a574213253af8908ec14c1ffb0dbc1de5e463"}
Jul 16 18:01:44.530921 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.530830    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:44.551557 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.551512    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k" podStartSLOduration=1.551497159 podStartE2EDuration="1.551497159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 18:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 18:01:44.551068392 +0000 UTC m=+1634.974798454" watchObservedRunningTime="2026-07-16 18:01:44.551497159 +0000 UTC m=+1634.975227218"
Jul 16 18:01:44.967796 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.967771    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4bnd_7956263e-465c-4266-b46d-e271490872ed/dns/0.log"
Jul 16 18:01:44.993851 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:44.993820    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z4bnd_7956263e-465c-4266-b46d-e271490872ed/kube-rbac-proxy/0.log"
Jul 16 18:01:45.047493 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:45.047464    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qzlzg_f3e969db-03bb-4ef2-8053-4e85ee82b946/dns-node-resolver/0.log"
Jul 16 18:01:45.653800 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:45.653766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-nx72m_5e2625d2-c2e2-4d64-a7b2-e5def0303940/node-ca/0.log"
Jul 16 18:01:46.586874 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:46.586842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-b8db8bc8d-dnsc4_90995984-c754-40fd-acb4-e505ab467da1/router/0.log"
Jul 16 18:01:46.969949 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:46.969871    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8rfjs_6c796012-9277-4e77-8226-60fd5be1dbd8/serve-healthcheck-canary/0.log"
Jul 16 18:01:47.446253 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:47.446214    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7mght_3dbcd895-ffcb-4eb3-be2b-871643f6110b/insights-operator/0.log"
Jul 16 18:01:47.446555 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:47.446532    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7mght_3dbcd895-ffcb-4eb3-be2b-871643f6110b/insights-operator/1.log"
Jul 16 18:01:47.555732 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:47.555700    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts6vm_b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed/kube-rbac-proxy/0.log"
Jul 16 18:01:47.581369 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:47.581344    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts6vm_b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed/exporter/0.log"
Jul 16 18:01:47.607066 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:47.607044    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-ts6vm_b6c7f738-ca53-4aa2-a2bf-5ff5e6db20ed/extractor/0.log"
Jul 16 18:01:49.880830 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:49.880804    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-fd65d7d6-7xg4w_0a1c60c2-1bc1-4056-8a05-ad70b5fc7650/server/0.log"
Jul 16 18:01:49.947640 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:49.947605    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-84b54b7f97-knwdl_9fbdcd3a-982e-4856-b420-c720bd6cbcee/manager/0.log"
Jul 16 18:01:50.168337 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:50.168255    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-scsz5_bfc329ba-0dbe-4ad5-b2af-906d8d05aa53/seaweedfs/0.log"
Jul 16 18:01:50.226351 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:50.226322    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-tls-serving-67b7477c6d-w8fc9_940bd3d7-fbda-4af0-b68c-4e96c9c21017/seaweedfs-tls-serving/0.log"
Jul 16 18:01:50.544861 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:50.544792    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-x48rf/perf-node-gather-daemonset-5vj2k"
Jul 16 18:01:56.608859 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.608829    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/kube-multus-additional-cni-plugins/0.log"
Jul 16 18:01:56.635338 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.635308    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/egress-router-binary-copy/0.log"
Jul 16 18:01:56.661916 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.661890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/cni-plugins/0.log"
Jul 16 18:01:56.687511 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.687494    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/bond-cni-plugin/0.log"
Jul 16 18:01:56.715771 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.715744    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/routeoverride-cni/0.log"
Jul 16 18:01:56.745331 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.745307    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/whereabouts-cni-bincopy/0.log"
Jul 16 18:01:56.772261 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:56.772238    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wlj77_30488cdf-7b40-4e46-ab07-b94cb7706253/whereabouts-cni/0.log"
Jul 16 18:01:57.065662 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:57.065624    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-vcz5q_06fccf7d-2fcb-47cf-b0f4-3d4f169a1254/kube-multus/0.log"
Jul 16 18:01:57.090268 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:57.090246    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48hrx_f861f763-4c0a-4bfa-9880-aabe7948600b/network-metrics-daemon/0.log"
Jul 16 18:01:57.115238 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:57.115206    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48hrx_f861f763-4c0a-4bfa-9880-aabe7948600b/kube-rbac-proxy/0.log"
Jul 16 18:01:58.352422 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.352393    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-controller/0.log"
Jul 16 18:01:58.374096 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.374071    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/0.log"
Jul 16 18:01:58.374235 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.374071    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovn-acl-logging/1.log"
Jul 16 18:01:58.394921 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.394896    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/kube-rbac-proxy-node/0.log"
Jul 16 18:01:58.421702 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.421678    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/kube-rbac-proxy-ovn-metrics/0.log"
Jul 16 18:01:58.445562 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.445535    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/northd/0.log"
Jul 16 18:01:58.472262 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.472238    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/nbdb/0.log"
Jul 16 18:01:58.499613 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.499590    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/sbdb/0.log"
Jul 16 18:01:58.542696 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:58.542668    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ktrzs_393a0ebb-5874-4cec-b039-b84de1526fb4/ovnkube-controller/0.log"
Jul 16 18:01:59.859517 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:01:59.859494    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-fqbvg_bb52c6e9-2de3-410b-ade0-87c4631fa276/network-check-target-container/0.log"
Jul 16 18:02:00.981632 ip-10-0-143-77 kubenswrapper[2576]: I0716 18:02:00.981602    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-8s7qq_96ac4f84-129c-48f7-bd5f-0f661b0f80b8/iptables-alerter/0.log"