Jul 02 18:28:26.115249 ip-10-0-130-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 02 18:28:26.115263 ip-10-0-130-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 02 18:28:26.115273 ip-10-0-130-27 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 02 18:28:26.115632 ip-10-0-130-27 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 02 18:28:35.639295 ip-10-0-130-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 02 18:28:35.639308 ip-10-0-130-27 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 935e5a7178924721b93514e8ae9cb708 --
Jul 02 18:30:23.560240 ip-10-0-130-27 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.927700    2553 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933805    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933819    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933824    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933830    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933835    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933839    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933843    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:23.935352 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933848    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933851    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933856    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933861    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933865    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933870    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933873    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933878    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933885    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933890    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933896    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933902    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933907    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933911    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933915    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933919    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933924    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933928    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933931    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:23.941688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933935    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933939    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933944    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933948    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933952    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933957    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933961    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933964    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933968    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933986    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933991    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933994    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.933999    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934003    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934007    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934011    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934015    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934019    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934023    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:23.942345 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934027    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934032    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934036    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934040    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934044    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934048    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934052    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934056    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934060    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934066    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934070    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934074    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934079    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934083    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934087    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934091    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934095    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934099    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934103    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:23.943211 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934107    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934111    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934115    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934119    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934123    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934126    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934130    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934134    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934138    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934141    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934145    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934149    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934152    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934156    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934160    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934163    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934167    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934170    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934174    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:23.943873 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934180    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934183    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934187    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934191    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934194    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934198    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934201    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934207    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934210    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934215    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934219    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934223    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934227    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934231    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934235    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.934239    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935256    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935265    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935270    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935275    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:23.944522 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935279    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935284    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935287    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935291    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935296    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935299    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935303    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935307    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935311    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935315    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935320    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935324    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935328    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935334    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935339    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935344    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935348    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935352    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935356    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935361    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:23.945147 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935365    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935369    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935373    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935377    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935381    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935384    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935389    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935393    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935397    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935401    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935404    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935408    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935414    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935418    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935423    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935427    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935431    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935434    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935438    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:23.945688 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935442    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935446    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935449    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935453    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935458    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935462    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935466    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935470    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935473    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935477    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935482    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935488    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935492    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935496    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935500    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935503    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935507    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935511    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935515    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935519    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:23.946227 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935524    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935527    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935531    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935535    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935539    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935543    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935546    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935550    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935554    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935558    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935563    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935567    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935570    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935574    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935578    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935583    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935594    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935597    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935601    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935606    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:23.946742 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935611    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935614    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935618    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935624    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935629    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935633    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935637    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935641    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935646    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935650    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935654    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935658    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935662    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935666    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935670    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.935673    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936182    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936190    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936194    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:23.947312 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936198    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936202    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936207    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936211    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936215    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936218    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936224    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936228    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936232    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936235    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936239    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936243    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936247    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936251    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936255    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936259    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936266    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936271    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936275    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936280    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:23.947835 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936284    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936288    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936292    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936296    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936300    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936304    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936309    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936313    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936316    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936320    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936324    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936328    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936332    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936336    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936342    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936347    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936352    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936357    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936361    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936365    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:23.948455 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936369    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936373    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936377    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936380    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936384    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936388    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936391    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936395    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936399    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936402    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936406    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936413    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936417    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936420    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936425    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936429    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936433    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936437    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936440    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:23.948966 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936444    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936448    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936451    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936455    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936459    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936463    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936468    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936472    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936477    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936481    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936485    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936488    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936492    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936496    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936500    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936503    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936507    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936511    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936514    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936518    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:23.949511 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936522    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936525    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936530    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936533    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936539    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936542    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936546    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936550    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936554    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936558    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936562    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936565    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936569    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936573    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936577    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936580    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.936584    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936672    2553 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936683    2553 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936691    2553 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936697    2553 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 18:30:24.070356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936703    2553 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 18:30:24.006663 ip-10-0-130-27 systemd[1]: Started Kubernetes Kubelet.
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936708    2553 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936714    2553 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936720    2553 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936724    2553 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936729    2553 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936734    2553 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936738    2553 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936743    2553 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936747    2553 flags.go:64] FLAG: --cgroup-root=""
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936752    2553 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936756    2553 flags.go:64] FLAG: --client-ca-file=""
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936760    2553 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936765    2553 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936771    2553 flags.go:64] FLAG: --cluster-domain=""
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936777    2553 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936782    2553 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936786    2553 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936791    2553 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936800    2553 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936804    2553 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936808    2553 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936813    2553 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936818    2553 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 18:30:24.071467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936822    2553 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936826    2553 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936831    2553 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936835    2553 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936841    2553 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936847    2553 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936851    2553 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936855    2553 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936859    2553 flags.go:64] FLAG: --enable-server="true"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936865    2553 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936871    2553 flags.go:64] FLAG: --event-burst="100"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936875    2553 flags.go:64] FLAG: --event-qps="50"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936879    2553 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936883    2553 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936888    2553 flags.go:64] FLAG: --eviction-hard=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936893    2553 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936897    2553 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936901    2553 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936906    2553 flags.go:64] FLAG: --eviction-soft=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936910    2553 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936914    2553 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936919    2553 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936922    2553 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936928    2553 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936932    2553 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 18:30:24.072300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936937    2553 flags.go:64] FLAG: --feature-gates=""
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936943    2553 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936947    2553 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936951    2553 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936956    2553 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936960    2553 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936964    2553 flags.go:64] FLAG: --help="false"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936984    2553 flags.go:64] FLAG: --hostname-override="ip-10-0-130-27.ec2.internal"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936989    2553 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936993    2553 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.936998    2553 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937003    2553 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937012    2553 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937016    2553 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937020    2553 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937024    2553 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937029    2553 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937034    2553 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937038    2553 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937043    2553 flags.go:64] FLAG: --kube-reserved=""
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937047    2553 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937051    2553 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937056    2553 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937060    2553 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 18:30:24.073127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937064    2553 flags.go:64] FLAG: --lock-file=""
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937068    2553 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937073    2553 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937077    2553 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937085    2553 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937089    2553 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937094    2553 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937098    2553 flags.go:64] FLAG: --logging-format="text"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937102    2553 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937107    2553 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937111    2553 flags.go:64] FLAG: --manifest-url=""
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937115    2553 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937121    2553 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937126    2553 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937131    2553 flags.go:64] FLAG: --max-pods="110"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937136    2553 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937140    2553 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937145    2553 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937149    2553 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937153    2553 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937159    2553 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937164    2553 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937173    2553 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937178    2553 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937182    2553 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 18:30:24.073858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937186    2553 flags.go:64] FLAG: --pod-cidr=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937191    2553 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937198    2553 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937202    2553 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937206    2553 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937210    2553 flags.go:64] FLAG: --port="10250"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937214    2553 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937219    2553 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02f85166cf76d3903"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937224    2553 flags.go:64] FLAG: --qos-reserved=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937228    2553 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937232    2553 flags.go:64] FLAG: --register-node="true"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937236    2553 flags.go:64] FLAG: --register-with-taints=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937242    2553 flags.go:64] FLAG: --registry-burst="10"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937247    2553 flags.go:64] FLAG: --registry-qps="5"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937251    2553 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937255    2553 flags.go:64] FLAG: --reserved-memory=""
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937260    2553 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937265    2553 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937269    2553 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937273    2553 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937277    2553 flags.go:64] FLAG: --runonce="false"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937282    2553 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937286    2553 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937290    2553 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937295    2553 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 18:30:24.074667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937299    2553 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937304    2553 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937308    2553 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937314    2553 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937318    2553 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937322    2553 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937326    2553 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937331    2553 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937335    2553 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937340    2553 flags.go:64] FLAG: --system-cgroups=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937344    2553 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937352    2553 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937355    2553 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937360    2553 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937370    2553 flags.go:64] FLAG: --tls-min-version=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937374    2553 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937378    2553 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937382    2553 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937386    2553 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937391    2553 flags.go:64] FLAG: --v="2"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937398    2553 flags.go:64] FLAG: --version="false"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937403    2553 flags.go:64] FLAG: --vmodule=""
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937408    2553 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937413    2553 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937571    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:24.075805 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937577    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937582    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937586    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937590    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937594    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937598    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937602    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937606    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937611    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937616    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937622    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937626    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937630    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937634    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937638    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937642    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937646    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937651    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937655    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937658    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:24.076569 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937662    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937666    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937671    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937675    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937679    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937683    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937686    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937692    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937696    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937699    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937703    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937707    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937711    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937715    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937719    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937723    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937727    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937732    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937735    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937739    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:24.248036 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937743    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937747    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937753    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937757    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937761    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937764    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937769    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937772    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937776    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937779    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937784    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937788    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937792    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937795    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937799    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937803    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937807    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937811    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937814    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:24.249712 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937820    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937823    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937827    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937831    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937835    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937838    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937842    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937846    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937850    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937854    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937858    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937862    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937866    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937870    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937873    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937878    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937883    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937886    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937892    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937896    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:24.250416 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937901    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937905    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937909    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937913    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937917    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937921    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937925    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937929    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937932    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937936    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937940    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937943    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937949    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937953    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937956    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937960    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937963    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937967    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:24.251130 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.937987    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.937995    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.948714    2553 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.948727    2553 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.949490    2553 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.949497    2553 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949554    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949559    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949563    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949566    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949570    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949573    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949576    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949580    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:24.251728 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949584    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949587    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949589    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949592    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949595    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949598    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949601    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949604    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949606    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949610    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949612    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949615    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949618    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949621    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949624    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949627    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949630    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949632    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949635    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949637    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:24.252231 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949641    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949645    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949648    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949650    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949653    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949656    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949659    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949661    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949664    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949667    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949669    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949672    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949674    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949677    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949680    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949682    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949685    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949688    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949691    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949695    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:24.252830 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949698    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949702    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949704    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949707    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949710    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949713    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949716    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949719    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949721    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949724    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949726    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949729    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949731    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949734    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949737    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949739    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949742    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949744    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949746    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949749    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:24.253469 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949752    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949754    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949757    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949759    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949762    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949764    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949767    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949769    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949772    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949775    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949778    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949781    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949783    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949786    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949788    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949791    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949794    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949797    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949799    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:24.254075 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949802    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949806    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949809    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949812    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949815    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949818    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949820    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949823    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949826    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949829    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949831    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949835    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.949839    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949942    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949946    2553 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 18:30:24.254683 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949949    2553 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949952    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949955    2553 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949958    2553 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949961    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949964    2553 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949967    2553 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949985    2553 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949988    2553 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949992    2553 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949995    2553 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.949998    2553 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950000    2553 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950003    2553 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950006    2553 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950008    2553 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950011    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950013    2553 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950017    2553 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 18:30:24.255162 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950022    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950024    2553 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950027    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950029    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950032    2553 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950035    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950038    2553 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950040    2553 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950043    2553 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950046    2553 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950048    2553 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950051    2553 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950054    2553 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950056    2553 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950059    2553 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950061    2553 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950064    2553 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950066    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950069    2553 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950071    2553 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 18:30:24.255748 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950074    2553 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950076    2553 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950079    2553 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950082    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950085    2553 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950087    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950090    2553 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950092    2553 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950095    2553 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950097    2553 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950101    2553 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950105    2553 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950108    2553 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950111    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950114    2553 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950116    2553 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950119    2553 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950121    2553 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950124    2553 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 18:30:24.256366 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950127    2553 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950129    2553 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950133    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950135    2553 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950137    2553 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950140    2553 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950143    2553 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950145    2553 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950148    2553 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950150    2553 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950153    2553 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950155    2553 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950158    2553 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950161    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950163    2553 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950166    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950168    2553 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950172    2553 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950174    2553 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950177    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 18:30:24.257116 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950180    2553 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950184    2553 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950187    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950190    2553 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950192    2553 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950195    2553 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950197    2553 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950200    2553 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950202    2553 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950205    2553 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950207    2553 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950210    2553 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950212    2553 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950215    2553 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950217    2553 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950220    2553 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950222    2553 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950225    2553 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 18:30:24.257743 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:23.950227    2553 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.950232    2553 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.950318    2553 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.952945    2553 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.954211    2553 server.go:1019] "Starting client certificate rotation"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.954303    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.954336    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.957676    2553 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.962618    2553 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.974872    2553 log.go:25] "Validated CRI v1 runtime API"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.977178    2553 log.go:25] "Validated CRI v1 image API"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.978433    2553 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 18:30:24.258454 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.980734    2553 fs.go:135] Filesystem UUIDs: map[48eef0d2-f638-40b2-94bf-d11b26168ba1:/dev/nvme0n1p4 6fb1755e-a634-4958-a6ca-33931b8f3a29:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 02 18:30:24.258842 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.980748    2553 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.985928    2553 manager.go:216] Machine: {Timestamp:2026-07-02 18:30:23.984797208 +0000 UTC m=+0.314046823 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098347 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28eafe90c8a10858bced033624c229 SystemUUID:ec28eafe-90c8-a108-58bc-ed033624c229 BootID:935e5a71-7892-4721-b935-14e8ae9cb708 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:d7:cf:e2:c3:d1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:d7:cf:e2:c3:d1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:15:d0:df:9d:c6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.986043    2553 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.986146    2553 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987049    2553 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987071    2553 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987831    2553 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987841    2553 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 18:30:24.258954 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987849    2553 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.987868    2553 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.989197    2553 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.990109    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.991994    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992114    2553 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992142    2553 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992155    2553 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992169    2553 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992177    2553 kubelet.go:398] "Adding apiserver pod source"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992186    2553 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992561    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.992578    2553 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.994617    2553 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.996381    2553 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.996414    2553 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997336    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997348    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997356    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997365    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997377    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997387    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997399    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997406    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997420    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997432    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997442    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.997447    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.998007    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:23.998013    2553 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.001221    2553 server.go:1268] "Started kubelet"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.005087    2553 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.005538    2553 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.005927    2553 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.006014    2553 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.006351    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.006396    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.007653    2553 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 18:30:24.259305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.008358    2553 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.010617    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-27.ec2.internal.18be8c895810c2b3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-27.ec2.internal,UID:ip-10-0-130-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-27.ec2.internal,},FirstTimestamp:2026-07-02 18:30:24.001196723 +0000 UTC m=+0.330446338,LastTimestamp:2026-07-02 18:30:24.001196723 +0000 UTC m=+0.330446338,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-27.ec2.internal,}"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.012954    2553 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.013841    2553 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.013849    2553 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.013931    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-27.ec2.internal\" not found"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014483    2553 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014501    2553 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014487    2553 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014570    2553 factory.go:55] Registering systemd factory
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014584    2553 factory.go:223] Registration of the systemd container factory successfully
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014602    2553 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014611    2553 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014775    2553 factory.go:153] Registering CRI-O factory
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014786    2553 factory.go:223] Registration of the crio container factory successfully
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014832    2553 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014856    2553 factory.go:103] Registering Raw factory
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.014872    2553 manager.go:1196] Started watching for new ooms in manager
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.015281    2553 manager.go:349] Starting recovery of all containers
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.022729    2553 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.022811    2553 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.030563    2553 manager.go:354] Recovery completed
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.031313    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqknd"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.034716    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.036665    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.036688    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.036698    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037237    2553 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037246    2553 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037263    2553 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037246    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037932    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.260556 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037958    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.037968    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.038318    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nqknd"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.038670    2553 policy_none.go:49] "None policy: Start"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.038698    2553 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.038717    2553 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.038410    2553 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-27.ec2.internal.18be8c895a2e213f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-27.ec2.internal,UID:ip-10-0-130-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-27.ec2.internal,},FirstTimestamp:2026-07-02 18:30:24.036675903 +0000 UTC m=+0.365925517,LastTimestamp:2026-07-02 18:30:24.036675903 +0000 UTC m=+0.365925517,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-27.ec2.internal,}"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.039731    2553 policy_none.go:47] "Start"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.090556    2553 manager.go:337] "Starting Device Plugin manager"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.090751    2553 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.090762    2553 server.go:87] "Starting device plugin registration server"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.091008    2553 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.091023    2553 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.091108    2553 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.091189    2553 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.091201    2553 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.091695    2553 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.091759    2553 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-27.ec2.internal\" not found"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.154924    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.156069    2553 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.156102    2553 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.156127    2553 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.156160    2553 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.158633    2553 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.191823    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.192630    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.192654    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.192664    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.192687    2553 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-27.ec2.internal"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.200431    2553 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-27.ec2.internal"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.200449    2553 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-27.ec2.internal\": node \"ip-10-0-130-27.ec2.internal\" not found"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:24.225687    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-27.ec2.internal\" not found"
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.256450    2553 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"]
Jul 02 18:30:24.264292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.256529    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.258196    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.258227    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.258237    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.259543    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.259666    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.259697    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260330    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260358    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260376    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260330    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260434    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.260446    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.261829    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.261850    2553 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.262933    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.262957    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 18:30:24.265589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:24.262970    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeHasSufficientPID"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.345464    2553 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:25.345510    2553 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-27.ec2.internal\" not found"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.345509    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 18:25:25 +0000 UTC" deadline="2027-12-28 01:45:08.2246834 +0000 UTC"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.345545    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13039h14m42.879143181s"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.359989    2553 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:25.383426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.360403    2553 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:25.671824 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.671797    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.671911 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.671888    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.671964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.671921    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab6471031b369a203d6efad60cbe0235-config\") pod \"kube-apiserver-proxy-ip-10-0-130-27.ec2.internal\" (UID: \"ab6471031b369a203d6efad60cbe0235\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.671964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.671944    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.679531 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.679513    2553 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 18:30:25.685554 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.685470    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 18:30:25.686490 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.686467    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.690336 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.690321    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.694938 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.694919    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 18:30:25.695154 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.695141    2553 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.703020 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.703005    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 18:30:25.703094 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:25.703042    2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.703253 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.703239    2553 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 18:30:25.703290 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:25.703275    2553 kubelet.go:3250] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-27.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773508 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773483    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773607 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773607 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773543    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab6471031b369a203d6efad60cbe0235-config\") pod \"kube-apiserver-proxy-ip-10-0-130-27.ec2.internal\" (UID: \"ab6471031b369a203d6efad60cbe0235\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773607 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773564    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773705 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773604    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/95f2247e4e78d7307ba66f65eeb96d48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal\" (UID: \"95f2247e4e78d7307ba66f65eeb96d48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:25.773705 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:25.773646    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ab6471031b369a203d6efad60cbe0235-config\") pod \"kube-apiserver-proxy-ip-10-0-130-27.ec2.internal\" (UID: \"ab6471031b369a203d6efad60cbe0235\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:26.006389 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.006360    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:26.008671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.008651    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal"
Jul 02 18:30:26.347800 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.347698    2553 apiserver.go:52] "Watching apiserver"
Jul 02 18:30:26.347800 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.347712    2553 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 18:30:26.356863 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.356841    2553 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 18:30:26.357929 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.357909    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-27jxq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal","openshift-multus/multus-wqwvf","openshift-network-diagnostics/network-check-target-kvrxf","openshift-network-operator/iptables-alerter-6cxpp","openshift-ovn-kubernetes/ovnkube-node-n2nk2","kube-system/global-pull-secret-syncer-d5rdt","kube-system/konnectivity-agent-f6qtv","kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9","openshift-dns/node-resolver-fxjbg","openshift-multus/multus-additional-cni-plugins-vmncx","openshift-multus/network-metrics-daemon-48vdd","openshift-cluster-node-tuning-operator/tuned-wx2fv"]
Jul 02 18:30:26.359733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.359716    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.359821 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.359801    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:26.360351 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.360332    2553 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 18:30:26.360811 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.360793    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.360887 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.360867    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:26.361923 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.361909    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.363186 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.363162    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:26.363272 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.363211    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:26.364281 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.364262    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.364991 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.364971    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 18:30:26.365092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.365077    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.365161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.365149    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 18:30:26.365261 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.365244    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8jzq6\""
Jul 02 18:30:26.365334 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.365320    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.365648 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.365631    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.366858 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.366840    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.367133 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.367102    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.367281 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.367265    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 18:30:26.367330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.367318    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f6zrh\""
Jul 02 18:30:26.367667 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.367654    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.368062 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368049    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 18:30:26.368281 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368267    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.368652 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368634    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7whtg\""
Jul 02 18:30:26.368729 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368663    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.368781 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368735    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 18:30:26.368781 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368749    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.368896 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.368873    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 18:30:26.369597 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.369576    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.369718 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.369700    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 18:30:26.369958 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.369940    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.370116 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.370067    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kvmkg\""
Jul 02 18:30:26.370241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.370176    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.370741 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.370725    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.371014 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.370991    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.371178 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.371153    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.371799 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.371782    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 18:30:26.371859 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.371789    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mrffm\""
Jul 02 18:30:26.372225 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.372210    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 18:30:26.372779 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.372687    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 18:30:26.372779 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.372695    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-gx7v9\""
Jul 02 18:30:26.373033 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.373000    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.374410 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.373992    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.374410 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.374298    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.374410 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.374302    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 18:30:26.374787 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.374707    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-75ksx\""
Jul 02 18:30:26.375702 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.375601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.376337 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.376312    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 18:30:26.376443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.376402    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-h8bq9\""
Jul 02 18:30:26.376494 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.376464    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 18:30:26.378094 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.378077    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 18:30:26.378519 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.378503    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wkj8f\""
Jul 02 18:30:26.378744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.378728    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 18:30:26.388982 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.388950    2553 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cv42t"
Jul 02 18:30:26.396059 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.396028    2553 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cv42t"
Jul 02 18:30:26.415801 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.415781    2553 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 18:30:26.477480 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477461    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-node-log\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477555 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477488    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-lib-modules\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.477555 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477506    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-var-lib-kubelet\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.477555 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477531    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-daemon-config\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477561    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-etc-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477580    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzvrb\" (UniqueName: \"kubernetes.io/projected/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-kube-api-access-lzvrb\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477598    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477614    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-system-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477636    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-bin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.477653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477652    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-kubelet\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477669    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-system-cni-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477685    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-bin\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477742    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-netns\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477759    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477776    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cnibin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.477819 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477800    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-conf-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477822    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-systemd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477837    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477858    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf9dr\" (UniqueName: \"kubernetes.io/projected/86a22131-0126-4643-a945-41cbd09417dc-kube-api-access-nf9dr\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477881    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-os-release\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477898    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-etc-kubernetes\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477913    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-tuned\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477928    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.478016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.477952    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39829f56-6d6f-41cf-9d77-cc91777211e4-agent-certs\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-slash\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478060    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-var-lib-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478078    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-kubernetes\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478095    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3da5140-4064-4577-8191-4667846a5726-tmp-dir\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478111    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc54bf1-8835-4198-91aa-4a09c593c2df-host\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478138    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478170    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfpjg\" (UniqueName: \"kubernetes.io/projected/8eafaad5-84d2-4376-971b-e5e2b477e146-kube-api-access-vfpjg\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478194    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-config\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478231    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-sys-fs\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.478266 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478250    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-hostroot\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478272    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-kubelet-config\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478295    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cni-binary-copy\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478315    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-os-release\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478334    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-dbus\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478356    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478377    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-conf\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478392    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-tmp\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478414    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4nxw\" (UniqueName: \"kubernetes.io/projected/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-kube-api-access-p4nxw\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478453    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478478    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478494    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-k8s-cni-cncf-io\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbt46\" (UniqueName: \"kubernetes.io/projected/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-kube-api-access-pbt46\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478542    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-kubelet\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478561    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-ovn\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478586    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovn-node-metrics-cert\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478601    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-sys\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478625    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39829f56-6d6f-41cf-9d77-cc91777211e4-konnectivity-ca\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478641    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478671    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eafaad5-84d2-4376-971b-e5e2b477e146-host-slash\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478693    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-log-socket\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478721    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-modprobe-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478764    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478790    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-registration-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478810    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4srcl\" (UniqueName: \"kubernetes.io/projected/4c19af73-31a8-4907-a934-380c2bcf71d7-kube-api-access-4srcl\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478831    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-multus\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478846    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3da5140-4064-4577-8191-4667846a5726-hosts-file\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.478904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478868    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-netns\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478884    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc7nv\" (UniqueName: \"kubernetes.io/projected/9adbcc24-afe2-400e-8524-fad68fdc6797-kube-api-access-fc7nv\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478920    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-systemd-units\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478946    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-netd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478961    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-env-overrides\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.478998    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysconfig\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479015    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-cnibin\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479045    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-script-lib\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479069    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eafaad5-84d2-4376-971b-e5e2b477e146-iptables-alerter-script\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479089    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479104    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-host\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479118    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-socket-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.479305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479136    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8pq6\" (UniqueName: \"kubernetes.io/projected/a3da5140-4064-4577-8191-4667846a5726-kube-api-access-t8pq6\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479157    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-socket-dir-parent\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479205    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc54bf1-8835-4198-91aa-4a09c593c2df-serviceca\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-systemd\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479247    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-run\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479269    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-device-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479292    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-multus-certs\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479308    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h865m\" (UniqueName: \"kubernetes.io/projected/edc54bf1-8835-4198-91aa-4a09c593c2df-kube-api-access-h865m\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.479725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.479338    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.579536 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579499    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-tmp\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.579536 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579532    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4nxw\" (UniqueName: \"kubernetes.io/projected/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-kube-api-access-p4nxw\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579550    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579569    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579588    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-k8s-cni-cncf-io\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579613    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbt46\" (UniqueName: \"kubernetes.io/projected/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-kube-api-access-pbt46\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579636    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-kubelet\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-ovn\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.579672    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:26.579746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579693    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-k8s-cni-cncf-io\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579746    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-kubelet\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579764    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-ovn\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579676    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovn-node-metrics-cert\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.579781    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:27.079730735 +0000 UTC m=+3.408980357 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579830    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-sys\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579838    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579859    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39829f56-6d6f-41cf-9d77-cc91777211e4-konnectivity-ca\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579879    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-sys\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579886    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579912    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eafaad5-84d2-4376-971b-e5e2b477e146-host-slash\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579955    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8eafaad5-84d2-4376-971b-e5e2b477e146-host-slash\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-log-socket\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-modprobe-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.579995    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-log-socket\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580040    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580049    2553 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580066    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-registration-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580130    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-registration-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580130    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-modprobe-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4srcl\" (UniqueName: \"kubernetes.io/projected/4c19af73-31a8-4907-a934-380c2bcf71d7-kube-api-access-4srcl\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580193    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-d\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580186    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-multus\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580233    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-multus\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580238    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3da5140-4064-4577-8191-4667846a5726-hosts-file\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580265    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-netns\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580300    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc7nv\" (UniqueName: \"kubernetes.io/projected/9adbcc24-afe2-400e-8524-fad68fdc6797-kube-api-access-fc7nv\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580326    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-systemd-units\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580335    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3da5140-4064-4577-8191-4667846a5726-hosts-file\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.580846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580351    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-netns\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580352    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-netd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580389    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-netd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-env-overrides\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580433    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysconfig\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580455    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-systemd-units\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580504    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysconfig\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580460    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-cnibin\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580524    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-cnibin\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580549    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-binary-copy\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580572    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/39829f56-6d6f-41cf-9d77-cc91777211e4-konnectivity-ca\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580581    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-script-lib\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580632    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eafaad5-84d2-4376-971b-e5e2b477e146-iptables-alerter-script\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.581591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580722    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580726    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-host\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580760    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-socket-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580766    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-host\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580784    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580790    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8pq6\" (UniqueName: \"kubernetes.io/projected/a3da5140-4064-4577-8191-4667846a5726-kube-api-access-t8pq6\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580911    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-env-overrides\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.580943    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-socket-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581016    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-socket-dir-parent\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581071    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc54bf1-8835-4198-91aa-4a09c593c2df-serviceca\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581097    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-systemd\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-run\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581145    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-device-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.582380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581148    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-script-lib\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581170    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-multus-certs\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581197    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h865m\" (UniqueName: \"kubernetes.io/projected/edc54bf1-8835-4198-91aa-4a09c593c2df-kube-api-access-h865m\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581210    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-run\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581224    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581239    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8eafaad5-84d2-4376-971b-e5e2b477e146-iptables-alerter-script\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581268    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-node-log\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.581293    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-lib-modules\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581327    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-var-lib-kubelet\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581334    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-systemd\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.581352    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:27.081335884 +0000 UTC m=+3.410585501 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581293    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-run-multus-certs\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581387    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-var-lib-kubelet\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.583836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581335    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-device-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-daemon-config\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581423    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-lib-modules\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581450    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-etc-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581469    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-node-log\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581481    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzvrb\" (UniqueName: \"kubernetes.io/projected/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-kube-api-access-lzvrb\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581508    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581509    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-etc-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581524    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/edc54bf1-8835-4198-91aa-4a09c593c2df-serviceca\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581616    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-etc-selinux\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581634    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-system-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581672    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-bin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581698    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-kubelet\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581742    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-system-cni-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581741    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-cni-bin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581765    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-system-cni-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581797    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-bin\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581802    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-system-cni-dir\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581746    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-host-var-lib-kubelet\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581825    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581858    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-netns\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581866    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-socket-dir-parent\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581869    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-cni-bin\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581887    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581891    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-daemon-config\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581922    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-netns\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581954    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cnibin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581982    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-conf-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582032    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-systemd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582044    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cnibin\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.584802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582056    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.581975    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-run-ovn-kubernetes\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582081    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf9dr\" (UniqueName: \"kubernetes.io/projected/86a22131-0126-4643-a945-41cbd09417dc-kube-api-access-nf9dr\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582107    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-os-release\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582124    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-systemd\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-etc-kubernetes\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582165    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-multus-conf-dir\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582180    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-tuned\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582183    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-os-release\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582127    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-run-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582206    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582246    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-etc-kubernetes\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39829f56-6d6f-41cf-9d77-cc91777211e4-agent-certs\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582326    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582329    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-slash\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582361    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-host-slash\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582382    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-var-lib-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582409    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-kubernetes\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582432    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3da5140-4064-4577-8191-4667846a5726-tmp-dir\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582451    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc54bf1-8835-4198-91aa-4a09c593c2df-host\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582433    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-var-lib-openvswitch\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582477    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfpjg\" (UniqueName: \"kubernetes.io/projected/8eafaad5-84d2-4376-971b-e5e2b477e146-kube-api-access-vfpjg\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582529    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/edc54bf1-8835-4198-91aa-4a09c593c2df-host\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582533    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-config\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582559    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-sys-fs\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582585    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-hostroot\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582609    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-kubelet-config\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582635    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cni-binary-copy\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.585664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582662    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-os-release\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582687    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-dbus\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582712    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582737    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-conf\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582791    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3da5140-4064-4577-8191-4667846a5726-tmp-dir\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582873    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-kubelet-config\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582874    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-sysctl-conf\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.582950    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9adbcc24-afe2-400e-8524-fad68fdc6797-os-release\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583014    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/07a2791d-f19a-4d3a-9a53-1ec166771a43-dbus\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583167    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583250    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4c19af73-31a8-4907-a934-380c2bcf71d7-sys-fs\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583254    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-hostroot\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583455    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovnkube-config\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583481    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9adbcc24-afe2-400e-8524-fad68fdc6797-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.586038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583552    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-kubernetes\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.586441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.583747    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-cni-binary-copy\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.586441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.584160    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-tmp\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.586441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.584363    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-ovn-node-metrics-cert\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.586441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.584374    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-etc-tuned\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.586441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.584767    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/39829f56-6d6f-41cf-9d77-cc91777211e4-agent-certs\") pod \"konnectivity-agent-f6qtv\" (UID: \"39829f56-6d6f-41cf-9d77-cc91777211e4\") " pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.596290 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.596268    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:26.596441 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.596298    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:26.596441 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.596315    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:26.596441 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:26.596376    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:27.096359299 +0000 UTC m=+3.425608923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:26.597148 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.597131    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbt46\" (UniqueName: \"kubernetes.io/projected/bd3cbe02-88e1-49b0-ae1e-063d5fceda7a-kube-api-access-pbt46\") pod \"multus-wqwvf\" (UID: \"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a\") " pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.597693 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.597673    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4nxw\" (UniqueName: \"kubernetes.io/projected/3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0-kube-api-access-p4nxw\") pod \"tuned-wx2fv\" (UID: \"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0\") " pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.597789 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.597714    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc7nv\" (UniqueName: \"kubernetes.io/projected/9adbcc24-afe2-400e-8524-fad68fdc6797-kube-api-access-fc7nv\") pod \"multus-additional-cni-plugins-vmncx\" (UID: \"9adbcc24-afe2-400e-8524-fad68fdc6797\") " pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.598398 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.598332    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4srcl\" (UniqueName: \"kubernetes.io/projected/4c19af73-31a8-4907-a934-380c2bcf71d7-kube-api-access-4srcl\") pod \"aws-ebs-csi-driver-node-d97q9\" (UID: \"4c19af73-31a8-4907-a934-380c2bcf71d7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.598759 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.598741    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzvrb\" (UniqueName: \"kubernetes.io/projected/2d6d1800-73c5-4a3f-951b-ab2b090cd3a7-kube-api-access-lzvrb\") pod \"ovnkube-node-n2nk2\" (UID: \"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7\") " pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.599146 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.599124    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h865m\" (UniqueName: \"kubernetes.io/projected/edc54bf1-8835-4198-91aa-4a09c593c2df-kube-api-access-h865m\") pod \"node-ca-27jxq\" (UID: \"edc54bf1-8835-4198-91aa-4a09c593c2df\") " pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.599299 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.599284    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfpjg\" (UniqueName: \"kubernetes.io/projected/8eafaad5-84d2-4376-971b-e5e2b477e146-kube-api-access-vfpjg\") pod \"iptables-alerter-6cxpp\" (UID: \"8eafaad5-84d2-4376-971b-e5e2b477e146\") " pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.599695 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.599678    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf9dr\" (UniqueName: \"kubernetes.io/projected/86a22131-0126-4643-a945-41cbd09417dc-kube-api-access-nf9dr\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:26.600015 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.599982    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8pq6\" (UniqueName: \"kubernetes.io/projected/a3da5140-4064-4577-8191-4667846a5726-kube-api-access-t8pq6\") pod \"node-resolver-fxjbg\" (UID: \"a3da5140-4064-4577-8191-4667846a5726\") " pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.693878 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.693847    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wqwvf"
Jul 02 18:30:26.701779 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.701753    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6cxpp"
Jul 02 18:30:26.725444 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.725418    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:26.744339 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.744310    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wx2fv"
Jul 02 18:30:26.747005 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.746977    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f2247e4e78d7307ba66f65eeb96d48.slice/crio-def1d675363925d3e1151d9b378bd10da870449b06a43dc198a8044db26cebbf WatchSource:0}: Error finding container def1d675363925d3e1151d9b378bd10da870449b06a43dc198a8044db26cebbf: Status 404 returned error can't find the container with id def1d675363925d3e1151d9b378bd10da870449b06a43dc198a8044db26cebbf
Jul 02 18:30:26.748106 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.748079    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab6471031b369a203d6efad60cbe0235.slice/crio-0e03823727bddd2502d04a1128306f2f4e4e909bf7fb02e3fb0390045f7e5eb7 WatchSource:0}: Error finding container 0e03823727bddd2502d04a1128306f2f4e4e909bf7fb02e3fb0390045f7e5eb7: Status 404 returned error can't find the container with id 0e03823727bddd2502d04a1128306f2f4e4e909bf7fb02e3fb0390045f7e5eb7
Jul 02 18:30:26.749447 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.749430    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6d1800_73c5_4a3f_951b_ab2b090cd3a7.slice/crio-33fc6611bbfc5eb0880a8b6243ca31baa326c78e42a8f4e6cb7e0f395b1793c2 WatchSource:0}: Error finding container 33fc6611bbfc5eb0880a8b6243ca31baa326c78e42a8f4e6cb7e0f395b1793c2: Status 404 returned error can't find the container with id 33fc6611bbfc5eb0880a8b6243ca31baa326c78e42a8f4e6cb7e0f395b1793c2
Jul 02 18:30:26.753949 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.753931    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:30:26.754460 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.754433    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3af6e73a_822a_4b59_8ad2_9bf6f5d61ad0.slice/crio-0ac8072764647e0b158eae044e96917ab541be4d475798c80e72f2af5011d195 WatchSource:0}: Error finding container 0ac8072764647e0b158eae044e96917ab541be4d475798c80e72f2af5011d195: Status 404 returned error can't find the container with id 0ac8072764647e0b158eae044e96917ab541be4d475798c80e72f2af5011d195
Jul 02 18:30:26.763433 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.763415    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27jxq"
Jul 02 18:30:26.769798 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.769774    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:26.771735 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.771638    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc54bf1_8835_4198_91aa_4a09c593c2df.slice/crio-ed2710352a8cfa6276d79bba9ad94c61aefb8b2f4d9a08abe4b5978ce08a1f46 WatchSource:0}: Error finding container ed2710352a8cfa6276d79bba9ad94c61aefb8b2f4d9a08abe4b5978ce08a1f46: Status 404 returned error can't find the container with id ed2710352a8cfa6276d79bba9ad94c61aefb8b2f4d9a08abe4b5978ce08a1f46
Jul 02 18:30:26.778521 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.778499    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39829f56_6d6f_41cf_9d77_cc91777211e4.slice/crio-0b5b49edb3525d19d82d1ce43fff61bdf86b9b0d17a854a343a808d2a15cd6a3 WatchSource:0}: Error finding container 0b5b49edb3525d19d82d1ce43fff61bdf86b9b0d17a854a343a808d2a15cd6a3: Status 404 returned error can't find the container with id 0b5b49edb3525d19d82d1ce43fff61bdf86b9b0d17a854a343a808d2a15cd6a3
Jul 02 18:30:26.789628 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.789602    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9"
Jul 02 18:30:26.796429 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.796408    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c19af73_31a8_4907_a934_380c2bcf71d7.slice/crio-b0dbf461fc7e95503f02875f104e93c09f087aca728c532a9b82f9f22a5647c9 WatchSource:0}: Error finding container b0dbf461fc7e95503f02875f104e93c09f087aca728c532a9b82f9f22a5647c9: Status 404 returned error can't find the container with id b0dbf461fc7e95503f02875f104e93c09f087aca728c532a9b82f9f22a5647c9
Jul 02 18:30:26.803532 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.803508    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-fxjbg"
Jul 02 18:30:26.808530 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:26.808515    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vmncx"
Jul 02 18:30:26.810263 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.810224    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3da5140_4064_4577_8191_4667846a5726.slice/crio-abc613f1bd1f7a59f3286ba05a635c0df46c6c7721eb8f6aeddf12eb8402dcf6 WatchSource:0}: Error finding container abc613f1bd1f7a59f3286ba05a635c0df46c6c7721eb8f6aeddf12eb8402dcf6: Status 404 returned error can't find the container with id abc613f1bd1f7a59f3286ba05a635c0df46c6c7721eb8f6aeddf12eb8402dcf6
Jul 02 18:30:26.816425 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:26.816404    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adbcc24_afe2_400e_8524_fad68fdc6797.slice/crio-29490d242d51018abd4f28e1ed95c4bc9f53e7de3266e48645652e1ee5ee9213 WatchSource:0}: Error finding container 29490d242d51018abd4f28e1ed95c4bc9f53e7de3266e48645652e1ee5ee9213: Status 404 returned error can't find the container with id 29490d242d51018abd4f28e1ed95c4bc9f53e7de3266e48645652e1ee5ee9213
Jul 02 18:30:27.072307 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:30:27.072279    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3cbe02_88e1_49b0_ae1e_063d5fceda7a.slice/crio-cbfea34d7751ad97eef3cbe75950ad429f4e2793085788929eea0bbb197a774e WatchSource:0}: Error finding container cbfea34d7751ad97eef3cbe75950ad429f4e2793085788929eea0bbb197a774e: Status 404 returned error can't find the container with id cbfea34d7751ad97eef3cbe75950ad429f4e2793085788929eea0bbb197a774e
Jul 02 18:30:27.085770 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.085720    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:27.085896 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.085882    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:27.085953 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.085891    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:27.086008 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.085958    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:28.085938065 +0000 UTC m=+4.415187682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:27.086087 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.086014    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:27.086087 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.086068    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:28.086052808 +0000 UTC m=+4.415302427 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:27.186277 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.186239    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:27.186941 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.186605    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:27.186941 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.186638    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:27.186941 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.186654    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:27.186941 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:27.186717    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:28.186697678 +0000 UTC m=+4.515947302 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:27.371033 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.370925    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqwvf" event={"ID":"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a","Type":"ContainerStarted","Data":"cbfea34d7751ad97eef3cbe75950ad429f4e2793085788929eea0bbb197a774e"}
Jul 02 18:30:27.394113 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.394073    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerStarted","Data":"29490d242d51018abd4f28e1ed95c4bc9f53e7de3266e48645652e1ee5ee9213"}
Jul 02 18:30:27.398306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.398182    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:27 +0000 UTC" deadline="2027-12-19 21:22:51.209381064 +0000 UTC"
Jul 02 18:30:27.398306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.398234    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12842h52m23.811151205s"
Jul 02 18:30:27.400342 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.400306    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxjbg" event={"ID":"a3da5140-4064-4577-8191-4667846a5726","Type":"ContainerStarted","Data":"abc613f1bd1f7a59f3286ba05a635c0df46c6c7721eb8f6aeddf12eb8402dcf6"}
Jul 02 18:30:27.415407 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.415330    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9" event={"ID":"4c19af73-31a8-4907-a934-380c2bcf71d7","Type":"ContainerStarted","Data":"b0dbf461fc7e95503f02875f104e93c09f087aca728c532a9b82f9f22a5647c9"}
Jul 02 18:30:27.439892 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.439857    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27jxq" event={"ID":"edc54bf1-8835-4198-91aa-4a09c593c2df","Type":"ContainerStarted","Data":"ed2710352a8cfa6276d79bba9ad94c61aefb8b2f4d9a08abe4b5978ce08a1f46"}
Jul 02 18:30:27.445076 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.444981    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wx2fv" event={"ID":"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0","Type":"ContainerStarted","Data":"0ac8072764647e0b158eae044e96917ab541be4d475798c80e72f2af5011d195"}
Jul 02 18:30:27.453131 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.453065    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal" event={"ID":"95f2247e4e78d7307ba66f65eeb96d48","Type":"ContainerStarted","Data":"def1d675363925d3e1151d9b378bd10da870449b06a43dc198a8044db26cebbf"}
Jul 02 18:30:27.472714 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.472620    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6cxpp" event={"ID":"8eafaad5-84d2-4376-971b-e5e2b477e146","Type":"ContainerStarted","Data":"7ec35ebb0a2a003b5fdfae29e3dc0a55bf57c90ba1a97002d1d17a280cdca35e"}
Jul 02 18:30:27.491750 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.491719    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6qtv" event={"ID":"39829f56-6d6f-41cf-9d77-cc91777211e4","Type":"ContainerStarted","Data":"0b5b49edb3525d19d82d1ce43fff61bdf86b9b0d17a854a343a808d2a15cd6a3"}
Jul 02 18:30:27.513530 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.513184    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"33fc6611bbfc5eb0880a8b6243ca31baa326c78e42a8f4e6cb7e0f395b1793c2"}
Jul 02 18:30:27.521910 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:27.521833    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal" event={"ID":"ab6471031b369a203d6efad60cbe0235","Type":"ContainerStarted","Data":"0e03823727bddd2502d04a1128306f2f4e4e909bf7fb02e3fb0390045f7e5eb7"}
Jul 02 18:30:28.104293 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.104257    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:28.104484 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.104324    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:28.104484 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.104459    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:28.104606 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.104522    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:30.104501416 +0000 UTC m=+6.433751027 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:28.104606 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.104594    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:28.104724 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.104630    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:30.104617903 +0000 UTC m=+6.433867507 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:28.159753 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.159719    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:28.159925 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.159842    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:28.160282 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.160253    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:28.160400 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.160358    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:28.160471 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.160420    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:28.160527 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.160499    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:28.205288 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.205253    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:28.205493 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.205469    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:28.205579 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.205503    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:28.205579 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.205519    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:28.205687 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:28.205581    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:30.205560847 +0000 UTC m=+6.534810451 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:28.398459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.398416    2553 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 18:25:27 +0000 UTC" deadline="2027-12-25 20:30:36.384556551 +0000 UTC"
Jul 02 18:30:28.398459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:28.398458    2553 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12986h0m7.98610288s"
Jul 02 18:30:30.121581 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.121426    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:30.121581 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.121501    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:30.122185 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.121654    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:30.122185 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.121721    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:34.121701746 +0000 UTC m=+10.450951365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:30.122345 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.122189    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:30.122345 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.122254    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:34.122236678 +0000 UTC m=+10.451486282 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:30.159084 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.159031    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:30.159290 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.159224    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:30.159635 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.159612    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:30.159739 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.159716    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:30.159804 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.159771    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:30.159870 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.159847    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:30.222553 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:30.222513    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:30.222763 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.222718    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:30.222763 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.222738    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:30.222763 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.222753    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:30.222931 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:30.222815    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:34.222795382 +0000 UTC m=+10.552045002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:32.158876 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:32.158047    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:32.158876 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:32.158209    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:32.158876 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:32.158660    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:32.158876 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:32.158763    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:32.159523 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:32.159435    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:32.159585 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:32.159539    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.157724    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.157856    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.157958    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.157997    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.158037    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.158054    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158158    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158168    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158203    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:42.158187422 +0000 UTC m=+18.487437025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158259    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:34.158263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158283    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:42.158275829 +0000 UTC m=+18.487525430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:34.159338 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.158250    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:34.258793 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:34.258750    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:34.258996 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.258979    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:34.259097 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.259007    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:34.259097 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.259039    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:34.259203 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:34.259099    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:42.259080149 +0000 UTC m=+18.588329768 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:36.157137 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:36.157097    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:36.157639 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:36.157097    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:36.157639 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:36.157238    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:36.157639 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:36.157306    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:36.157639 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:36.157383    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:36.157639 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:36.157461    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:38.157103 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:38.157072    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:38.157486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:38.157108    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:38.157486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:38.157072    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:38.157486 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:38.157190    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:38.157486 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:38.157262    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:38.157486 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:38.157339    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:40.156732 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:40.156702    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:40.157216 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:40.156710    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:40.157216 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:40.156830    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:40.157216 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:40.156711    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:40.157216 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:40.156937    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:40.157216 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:40.156988    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:42.157159 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.157126    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:42.157159 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.157152    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:42.157689 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.157180    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:42.157689 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.157260    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:42.157689 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.157663    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:42.157860 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.157754    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:42.217877 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.217844    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:42.218076 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.217918    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:42.218076 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.218023    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:42.218076 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.218052    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:42.218214 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.218099    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:30:58.218078779 +0000 UTC m=+34.547328381 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:42.218214 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.218121    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:58.218110751 +0000 UTC m=+34.547360351 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:42.318661 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:42.318619    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:42.318825 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.318774    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:42.318825 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.318795    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:42.318825 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.318804    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:42.318954 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:42.318856    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:30:58.318840505 +0000 UTC m=+34.648090109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:44.157937 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:44.157904    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:44.158380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:44.158022    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:44.158380 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:44.158040    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:44.158380 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:44.158129    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:44.158380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:44.158207    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:44.158380 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:44.158312    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:46.156477 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:46.156440    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:46.156918 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:46.156449    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:46.156918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:46.156576    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:46.156918 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:46.156462    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:46.156918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:46.156641    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:46.156918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:46.156715    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:48.156860 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.156838    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:48.157312 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.156892    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:48.157312 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.156898    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:48.157312 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:48.156996    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:48.157312 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:48.157074    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:48.157312 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:48.157157    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:48.583744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.583664    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:30:48.584066 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584042    2553 generic.go:356] "Generic (PLEG): container finished" podID="2d6d1800-73c5-4a3f-951b-ab2b090cd3a7" containerID="8eb639d6e904dfd4d6f4c66373dd509c2708cb3c48bb5eddffece13c83bd70f1" exitCode=1
Jul 02 18:30:48.584125 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584105    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"36be94cdd03342dc15ae88b55e09bde477f518009fdbe398b3c7caca2cea6de0"}
Jul 02 18:30:48.584183 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584176    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"4eb5536a8cbb5ec47061d24a144f70a04526432423d0316a8683e5350f1f78bc"}
Jul 02 18:30:48.584237 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584219    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"5af5c79819d530d3c2bd153cc995dcc4feac9aa7c0c8325887eb6a002dc7a79b"}
Jul 02 18:30:48.584285 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584261    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"88a445c243a5c46a6a0f0be634737535c4973875f3a4dd80116127cccc13eaa5"}
Jul 02 18:30:48.584338 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584289    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerDied","Data":"8eb639d6e904dfd4d6f4c66373dd509c2708cb3c48bb5eddffece13c83bd70f1"}
Jul 02 18:30:48.584338 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.584320    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"0bd3ece65ba33c67761afbe7a6757630ec04dac3877f2666835f84f11c9831c0"}
Jul 02 18:30:48.585665 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.585640    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal" event={"ID":"ab6471031b369a203d6efad60cbe0235","Type":"ContainerStarted","Data":"98e55a0972cb140d2633b1fa63e9e7c20ba783ba104b4e14480cbf7ae5b3baa0"}
Jul 02 18:30:48.590478 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.590450    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wqwvf" event={"ID":"bd3cbe02-88e1-49b0-ae1e-063d5fceda7a","Type":"ContainerStarted","Data":"8a66c2279d92a2d779682037ea66ee29669db3905c898b0ebad5d7d1f34ee500"}
Jul 02 18:30:48.591949 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.591925    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wx2fv" event={"ID":"3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0","Type":"ContainerStarted","Data":"ad496fbdcebc4c6ad4aed1f253cba7ed918b78ec5fefcfa1e0ff2e81eb93d355"}
Jul 02 18:30:48.603181 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.603136    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-27.ec2.internal" podStartSLOduration=22.603124356 podStartE2EDuration="22.603124356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:48.602795736 +0000 UTC m=+24.932045361" watchObservedRunningTime="2026-07-02 18:30:48.603124356 +0000 UTC m=+24.932373979"
Jul 02 18:30:48.620764 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.620694    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wqwvf" podStartSLOduration=2.7973830299999998 podStartE2EDuration="23.620674347s" totalImagesPullingTime="20.823291317s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:48.620383178 +0000 UTC m=+24.949632802" watchObservedRunningTime="2026-07-02 18:30:48.620674347 +0000 UTC m=+24.949923971"
Jul 02 18:30:48.640820 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:48.640757    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wx2fv" podStartSLOduration=2.803420203 podStartE2EDuration="23.640745562s" totalImagesPullingTime="20.837325359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:48.640659378 +0000 UTC m=+24.969909003" watchObservedRunningTime="2026-07-02 18:30:48.640745562 +0000 UTC m=+24.969995185"
Jul 02 18:30:49.595952 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.595911    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27jxq" event={"ID":"edc54bf1-8835-4198-91aa-4a09c593c2df","Type":"ContainerStarted","Data":"881187f1e9210564c736ba989adcda7863c0a62d682d81330a2423ca035d522b"}
Jul 02 18:30:49.597763 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.597741    2553 generic.go:356] "Generic (PLEG): container finished" podID="95f2247e4e78d7307ba66f65eeb96d48" containerID="5b04f7fd068ddf1a44c9c76d71ef7b8e0a6f04b49022b4a143c8b57fc476c29e" exitCode=0
Jul 02 18:30:49.597885 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.597819    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal" event={"ID":"95f2247e4e78d7307ba66f65eeb96d48","Type":"ContainerDied","Data":"5b04f7fd068ddf1a44c9c76d71ef7b8e0a6f04b49022b4a143c8b57fc476c29e"}
Jul 02 18:30:49.599505 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.599478    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6cxpp" event={"ID":"8eafaad5-84d2-4376-971b-e5e2b477e146","Type":"ContainerStarted","Data":"3313fdd32e81ce8ebf3cb13045c1526fc0111f05de830b423333572631c9f251"}
Jul 02 18:30:49.601103 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.601080    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-f6qtv" event={"ID":"39829f56-6d6f-41cf-9d77-cc91777211e4","Type":"ContainerStarted","Data":"77e29bcead57dd715caaf001870087da96c8ab264af2aa327d67f3f59d4628a1"}
Jul 02 18:30:49.602741 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.602719    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="d2e25bd109dca4654b95bb74a0c6ea86856ce5184e88445b7a044bfc52019f5b" exitCode=0
Jul 02 18:30:49.602838 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.602809    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"d2e25bd109dca4654b95bb74a0c6ea86856ce5184e88445b7a044bfc52019f5b"}
Jul 02 18:30:49.604364 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.604340    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-fxjbg" event={"ID":"a3da5140-4064-4577-8191-4667846a5726","Type":"ContainerStarted","Data":"f3983de9cb3d498d82ee2e14cde7f3ec296d368017afe5df165616db7fcdddda"}
Jul 02 18:30:49.606111 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.606082    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9" event={"ID":"4c19af73-31a8-4907-a934-380c2bcf71d7","Type":"ContainerStarted","Data":"8bf61992cc4a2741386b40c21a5f59c1811fceabe83116eac212357676d5a4cf"}
Jul 02 18:30:49.652706 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.652657    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-fxjbg" podStartSLOduration=3.87729116 podStartE2EDuration="24.652643915s" totalImagesPullingTime="20.775352755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:49.6516237 +0000 UTC m=+25.980873322" watchObservedRunningTime="2026-07-02 18:30:49.652643915 +0000 UTC m=+25.981893537"
Jul 02 18:30:49.652878 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.652860    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-27jxq" podStartSLOduration=3.83959229 podStartE2EDuration="24.652855641s" totalImagesPullingTime="20.813263351s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:49.616355878 +0000 UTC m=+25.945605502" watchObservedRunningTime="2026-07-02 18:30:49.652855641 +0000 UTC m=+25.982105313"
Jul 02 18:30:49.739452 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.739412    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6cxpp" podStartSLOduration=4.376610555 podStartE2EDuration="24.739397044s" totalImagesPullingTime="20.362786489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:49.739070837 +0000 UTC m=+26.068320462" watchObservedRunningTime="2026-07-02 18:30:49.739397044 +0000 UTC m=+26.068646667"
Jul 02 18:30:49.756148 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:49.756100    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-f6qtv" podStartSLOduration=3.949355927 podStartE2EDuration="24.756081931s" totalImagesPullingTime="20.806726004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:49.755918673 +0000 UTC m=+26.085168297" watchObservedRunningTime="2026-07-02 18:30:49.756081931 +0000 UTC m=+26.085331556"
Jul 02 18:30:50.086639 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.086612    2553 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 18:30:50.137055 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.137023    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:50.137761 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.137739    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:50.157271 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.157245    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:50.157422 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.157245    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:50.157422 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:50.157354    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:50.157524 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:50.157473    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:50.157524 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.157245    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:50.157624 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:50.157599    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:50.360833 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.360636    2553 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T18:30:50.086634881Z","UUID":"00990aa8-ef9f-4da8-b288-45c73a8cf94d","Handler":null,"Name":"","Endpoint":""}
Jul 02 18:30:50.362408 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.362380    2553 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 18:30:50.362533 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.362421    2553 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 18:30:50.612211 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.612168    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal" event={"ID":"95f2247e4e78d7307ba66f65eeb96d48","Type":"ContainerStarted","Data":"98c2f18808db70999e52c68af854fd438084d5cd8cffcfc53e29d0b131723124"}
Jul 02 18:30:50.616514 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.616167    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9" event={"ID":"4c19af73-31a8-4907-a934-380c2bcf71d7","Type":"ContainerStarted","Data":"56a064475039c475bc2b94a8cf9e698bae273a2c779d9c4367419e630ae87b17"}
Jul 02 18:30:50.616859 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.616688    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:50.617188 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.617171    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-f6qtv"
Jul 02 18:30:50.626837 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:50.626794    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-27.ec2.internal" podStartSLOduration=24.626779125 podStartE2EDuration="24.626779125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:50.626578047 +0000 UTC m=+26.955827674" watchObservedRunningTime="2026-07-02 18:30:50.626779125 +0000 UTC m=+26.956028743"
Jul 02 18:30:51.621101 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:51.621063    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9" event={"ID":"4c19af73-31a8-4907-a934-380c2bcf71d7","Type":"ContainerStarted","Data":"bb6819a380b5f3025e6790be65a393c3ed3fb96c06d8fbfd73fdb77cf49fc3da"}
Jul 02 18:30:51.624848 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:51.624823    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:30:51.625325 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:51.625300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"732fc13f4ad5810a300783bce201ad45352b80aa2f39a0370116ec0d6d7ca032"}
Jul 02 18:30:51.639579 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:51.639532    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-d97q9" podStartSLOduration=3.763988924 podStartE2EDuration="26.639515305s" totalImagesPullingTime="22.875526381s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:51.637655233 +0000 UTC m=+27.966904856" watchObservedRunningTime="2026-07-02 18:30:51.639515305 +0000 UTC m=+27.968764926"
Jul 02 18:30:52.157104 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:52.157078    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:52.157271 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:52.157079    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:52.157271 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:52.157205    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:52.157366 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:52.157289    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:52.157366 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:52.157088    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:52.157448 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:52.157422    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:54.157316 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.157247    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:54.157922 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:54.157328    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:54.157922 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.157359    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:54.157922 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:54.157435    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:54.157922 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.157460    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:54.157922 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:54.157535    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:54.633212 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.633177    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="bc69e80aaecb63b56f93b6ad22a6955087115066721c8e62b4718d573d67d28a" exitCode=0
Jul 02 18:30:54.633407 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.633265    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"bc69e80aaecb63b56f93b6ad22a6955087115066721c8e62b4718d573d67d28a"}
Jul 02 18:30:54.636619 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.636581    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:30:54.637027 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.636893    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"f84597733689f7a04bdd172f9e7f8b525eb45462db32cd748526c3866fc6d29f"}
Jul 02 18:30:54.637190 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.637174    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:54.637248 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.637229    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:54.637312 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.637297    2553 scope.go:117] "RemoveContainer" containerID="8eb639d6e904dfd4d6f4c66373dd509c2708cb3c48bb5eddffece13c83bd70f1"
Jul 02 18:30:54.653039 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:54.653010    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:55.600836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.600777    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48vdd"]
Jul 02 18:30:55.601256 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.600989    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:55.601256 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:55.601087    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:55.604288 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.604262    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d5rdt"]
Jul 02 18:30:55.604412 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.604378    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:55.604490 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:55.604468    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:55.604791 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.604764    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvrxf"]
Jul 02 18:30:55.604893 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.604859    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:55.604998 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:55.604960    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:55.643048 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.643021    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:30:55.643668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.643639    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" event={"ID":"2d6d1800-73c5-4a3f-951b-ab2b090cd3a7","Type":"ContainerStarted","Data":"142192cc96ba665e777bdccbcb62faf425db984acc32e8dbf95430ca24497891"}
Jul 02 18:30:55.644005 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.643962    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:55.659263 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.659244    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:30:55.676719 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:55.676675    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2" podStartSLOduration=9.542614298 podStartE2EDuration="30.67666013s" totalImagesPullingTime="21.134045832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:30:55.676394044 +0000 UTC m=+32.005643696" watchObservedRunningTime="2026-07-02 18:30:55.67666013 +0000 UTC m=+32.005909754"
Jul 02 18:30:56.650965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:56.650923    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="83d447af709d733ee679a0e8c5006028cba7427aca75584e5fe3783aef7293d3" exitCode=0
Jul 02 18:30:56.651407 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:56.650986    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"83d447af709d733ee679a0e8c5006028cba7427aca75584e5fe3783aef7293d3"}
Jul 02 18:30:57.157036 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:57.157005    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:57.157226 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:57.157005    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:57.157226 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:57.157144    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:57.157226 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:57.157005    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:57.157370 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:57.157234    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:57.157370 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:57.157324    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:30:57.655424 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:57.655393    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerStarted","Data":"45dcc27229995b55b73eb6a0b3968abf18815b4064a7856cfb70c6fb664b9790"}
Jul 02 18:30:58.238080 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:58.238051    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:58.238246 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:58.238100    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:58.238246 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.238207    2553 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:58.238315 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.238262    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret podName:07a2791d-f19a-4d3a-9a53-1ec166771a43 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:30.238246568 +0000 UTC m=+66.567496169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret") pod "global-pull-secret-syncer-d5rdt" (UID: "07a2791d-f19a-4d3a-9a53-1ec166771a43") : object "kube-system"/"original-pull-secret" not registered
Jul 02 18:30:58.238315 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.238209    2553 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:58.238402 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.238328    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs podName:86a22131-0126-4643-a945-41cbd09417dc nodeName:}" failed. No retries permitted until 2026-07-02 18:31:30.238318299 +0000 UTC m=+66.567567899 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs") pod "network-metrics-daemon-48vdd" (UID: "86a22131-0126-4643-a945-41cbd09417dc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 18:30:58.338825 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:58.338790    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:58.338997 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.338912    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 18:30:58.338997 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.338927    2553 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 18:30:58.338997 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.338935    2553 projected.go:196] Error preparing data for projected volume kube-api-access-7l76f for pod openshift-network-diagnostics/network-check-target-kvrxf: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:58.338997 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:58.338994    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f podName:61507973-7cb0-44d4-850c-05b89d4c27fc nodeName:}" failed. No retries permitted until 2026-07-02 18:31:30.338968137 +0000 UTC m=+66.668217737 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7l76f" (UniqueName: "kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f") pod "network-check-target-kvrxf" (UID: "61507973-7cb0-44d4-850c-05b89d4c27fc") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 18:30:58.660168 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:58.660080    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="45dcc27229995b55b73eb6a0b3968abf18815b4064a7856cfb70c6fb664b9790" exitCode=0
Jul 02 18:30:58.660168 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:58.660147    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"45dcc27229995b55b73eb6a0b3968abf18815b4064a7856cfb70c6fb664b9790"}
Jul 02 18:30:59.156891 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:59.156689    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:30:59.157064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:59.156709    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:30:59.157064 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:59.157001    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvrxf" podUID="61507973-7cb0-44d4-850c-05b89d4c27fc"
Jul 02 18:30:59.157185 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:30:59.156717    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:30:59.157185 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:59.157095    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-48vdd" podUID="86a22131-0126-4643-a945-41cbd09417dc"
Jul 02 18:30:59.157185 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:30:59.157173    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-d5rdt" podUID="07a2791d-f19a-4d3a-9a53-1ec166771a43"
Jul 02 18:31:00.766648 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.766616    2553 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-27.ec2.internal" event="NodeReady"
Jul 02 18:31:00.767071 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.766772    2553 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 18:31:00.816225 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.815800    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-6h2w9"]
Jul 02 18:31:00.836989 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.836944    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-674c7d54dc-s8b5k"]
Jul 02 18:31:00.837124 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.837080    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.841574 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.841549    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 18:31:00.842499 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.842471    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 18:31:00.842620 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.842514    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.842620 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.842531    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-jg4hk\""
Jul 02 18:31:00.842620 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.842607    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.849695 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.849672    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 18:31:00.861164 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.861137    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:31:00.862193 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.862100    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.868723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.868701    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 02 18:31:00.868723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.868717    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.868908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.868759    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-29f52\""
Jul 02 18:31:00.868908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.868745    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 02 18:31:00.868908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.868800    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 02 18:31:00.869119 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.869095    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.869222 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.869095    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 02 18:31:00.887227 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.887162    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"]
Jul 02 18:31:00.887357 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.887337    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.891818 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.891789    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 18:31:00.892178 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.892158    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 18:31:00.892724 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.892701    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r2j6b\""
Jul 02 18:31:00.892857 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.892702    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 18:31:00.898726 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.898706    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 18:31:00.906656 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.906638    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"]
Jul 02 18:31:00.906787 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.906772    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:00.915566 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.915548    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 02 18:31:00.915772 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.915760    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wxn7t\""
Jul 02 18:31:00.915865 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.915850    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.917189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.917173    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 02 18:31:00.917269 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.917232    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.928785 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.928764    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"]
Jul 02 18:31:00.928901 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.928886    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"
Jul 02 18:31:00.931666 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.931647    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.931770 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.931728    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-hdvjn\""
Jul 02 18:31:00.932032 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.932016    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.950876 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.950854    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ctnnf"]
Jul 02 18:31:00.950998 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.950984    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:00.954426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.954410    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wtqjs\""
Jul 02 18:31:00.954870 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.954849    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 02 18:31:00.955168 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.955149    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.955259 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.955233    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.958988 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.958952    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmphr\" (UniqueName: \"kubernetes.io/projected/88ac6aaf-06fa-4909-ae0d-14b8849883b8-kube-api-access-vmphr\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959087 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959005    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wkl\" (UniqueName: \"kubernetes.io/projected/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-kube-api-access-g5wkl\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:00.959087 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959034    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-tmp\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959087 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959058    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959238 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959085    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959238 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959144    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:00.959238 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959169    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959238 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959212    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-default-certificate\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959236    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j85bv\" (UniqueName: \"kubernetes.io/projected/f0979127-7619-4133-a067-e6f9c0078edd-kube-api-access-j85bv\") pod \"volume-data-source-validator-5d456fdbcd-ftjdp\" (UID: \"f0979127-7619-4133-a067-e6f9c0078edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959301    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959326    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959353    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959376    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.959429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959462    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-snapshots\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959485    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-stats-auth\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959515    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88ac6aaf-06fa-4909-ae0d-14b8849883b8-serving-cert\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959542    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqgs\" (UniqueName: \"kubernetes.io/projected/97f2cfcc-f466-4c0f-a731-0adf39a75625-kube-api-access-rcqgs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959569    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959594    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959622    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.959727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.959646    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2hr\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:00.975810 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975792    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-6h2w9"]
Jul 02 18:31:00.975810 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975805    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975848    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"]
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975874    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975892    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"]
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975917    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674c7d54dc-s8b5k"]
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975935    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"]
Jul 02 18:31:00.975965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.975958    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hjr72"]
Jul 02 18:31:00.978506 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.978484    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 02 18:31:00.978736 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.978715    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 02 18:31:00.979330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.979300    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:00.979330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.979324    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.979637 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.979617    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lmkzs\""
Jul 02 18:31:00.986023 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.986003    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 02 18:31:00.991737 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.991720    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5wbwr"]
Jul 02 18:31:00.991848 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.991822    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:00.994838 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.994821    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 18:31:00.995029 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.995010    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 18:31:00.995419 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.995404    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-82lp6\""
Jul 02 18:31:00.995683 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:00.995544    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 18:31:01.009663 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.009638    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ctnnf"]
Jul 02 18:31:01.009754 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.009672    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjr72"]
Jul 02 18:31:01.009754 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.009696    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wbwr"]
Jul 02 18:31:01.009860 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.009774    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.013318 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.013299    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 18:31:01.013878 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.013852    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 18:31:01.014160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.014144    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xnjsw\""
Jul 02 18:31:01.060370 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.060489 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060380    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-tmp\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.060489 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060399    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.060489 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060444    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.060489 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060478    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060503    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vts5h\" (UniqueName: \"kubernetes.io/projected/703b350d-5221-4434-b101-fe395ca946db-kube-api-access-vts5h\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060530    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060552    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-config\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.060564    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060580    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060605    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.060640    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.560619902 +0000 UTC m=+37.889869521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:01.060694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060665    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060699    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060725    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-tmp\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060727    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060768    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/703b350d-5221-4434-b101-fe395ca946db-serving-cert\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060810    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-stats-auth\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060827    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2hr\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060843    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqgs\" (UniqueName: \"kubernetes.io/projected/97f2cfcc-f466-4c0f-a731-0adf39a75625-kube-api-access-rcqgs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060862    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060899    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmphr\" (UniqueName: \"kubernetes.io/projected/88ac6aaf-06fa-4909-ae0d-14b8849883b8-kube-api-access-vmphr\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060916    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wkl\" (UniqueName: \"kubernetes.io/projected/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-kube-api-access-g5wkl\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060931    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060953    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-config-volume\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.061245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.060999    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-tmp-dir\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061023    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b55c16-655f-41b4-aea8-ea598a609543-cert\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061042    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-default-certificate\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061067    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfqn\" (UniqueName: \"kubernetes.io/projected/70b55c16-655f-41b4-aea8-ea598a609543-kube-api-access-fsfqn\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061093    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cttc\" (UniqueName: \"kubernetes.io/projected/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-kube-api-access-5cttc\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061119    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-snapshots\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061140    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061165    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk8gl\" (UniqueName: \"kubernetes.io/projected/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-kube-api-access-zk8gl\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061192    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061262    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.061284    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061325    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061523    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.062244 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.061720    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.561702862 +0000 UTC m=+37.890952485 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061717    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.061996    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/88ac6aaf-06fa-4909-ae0d-14b8849883b8-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062208    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88ac6aaf-06fa-4909-ae0d-14b8849883b8-serving-cert\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062228    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.062303    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.562287744 +0000 UTC m=+37.891537348 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062341    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-trusted-ca\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062382    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j85bv\" (UniqueName: \"kubernetes.io/projected/f0979127-7619-4133-a067-e6f9c0078edd-kube-api-access-j85bv\") pod \"volume-data-source-validator-5d456fdbcd-ftjdp\" (UID: \"f0979127-7619-4133-a067-e6f9c0078edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062445    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.062577    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.062591    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.062646    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.56263074 +0000 UTC m=+37.891880343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:01.062832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.062773    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/88ac6aaf-06fa-4909-ae0d-14b8849883b8-snapshots\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.065787 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.065742    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/88ac6aaf-06fa-4909-ae0d-14b8849883b8-serving-cert\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.065871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.065821    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.065871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.065830    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-stats-auth\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.066370 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.066337    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.068756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.068736    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.071172 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.071138    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wkl\" (UniqueName: \"kubernetes.io/projected/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-kube-api-access-g5wkl\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.072147 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.072129    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmphr\" (UniqueName: \"kubernetes.io/projected/88ac6aaf-06fa-4909-ae0d-14b8849883b8-kube-api-access-vmphr\") pod \"insights-operator-5c847dcfc4-6h2w9\" (UID: \"88ac6aaf-06fa-4909-ae0d-14b8849883b8\") " pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.072615 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.072594    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j85bv\" (UniqueName: \"kubernetes.io/projected/f0979127-7619-4133-a067-e6f9c0078edd-kube-api-access-j85bv\") pod \"volume-data-source-validator-5d456fdbcd-ftjdp\" (UID: \"f0979127-7619-4133-a067-e6f9c0078edd\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"
Jul 02 18:31:01.072696 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.072649    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2hr\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.076326 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.076305    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-default-certificate\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.077820 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.077803    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqgs\" (UniqueName: \"kubernetes.io/projected/97f2cfcc-f466-4c0f-a731-0adf39a75625-kube-api-access-rcqgs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.084846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.084797    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:01.149361 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.149296    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9"
Jul 02 18:31:01.156543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.156518    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:31:01.156543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.156533    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:31:01.156883 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.156850    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:31:01.159756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.159714    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-86j9v\""
Jul 02 18:31:01.160145 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.160123    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 18:31:01.160242 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.160195    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vvptw\""
Jul 02 18:31:01.160242 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.160197    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 18:31:01.160358 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.160258    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 18:31:01.160358 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.160221    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 18:31:01.162757 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162738    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b55c16-655f-41b4-aea8-ea598a609543-cert\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.162856 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162780    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfqn\" (UniqueName: \"kubernetes.io/projected/70b55c16-655f-41b4-aea8-ea598a609543-kube-api-access-fsfqn\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.162856 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162809    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cttc\" (UniqueName: \"kubernetes.io/projected/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-kube-api-access-5cttc\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.162959 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162852    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk8gl\" (UniqueName: \"kubernetes.io/projected/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-kube-api-access-zk8gl\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.162959 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162880    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.162959 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162915    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-trusted-ca\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.163129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.162957    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.163129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163022    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vts5h\" (UniqueName: \"kubernetes.io/projected/703b350d-5221-4434-b101-fe395ca946db-kube-api-access-vts5h\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.163129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163052    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-config\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.163129 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.163078    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:01.163129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163108    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/703b350d-5221-4434-b101-fe395ca946db-serving-cert\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.163356 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.163140    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.663121745 +0000 UTC m=+37.992371365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:01.163356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163340    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-config-volume\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.163470 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163375    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-tmp-dir\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.163470 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.163449    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:01.163615 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.163508    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:01.663491407 +0000 UTC m=+37.992741027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:01.163780 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.163681    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-tmp-dir\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.164403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.164026    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-trusted-ca\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.164403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.164037    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-config-volume\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.164403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.164351    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/703b350d-5221-4434-b101-fe395ca946db-config\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.165630 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.165610    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/703b350d-5221-4434-b101-fe395ca946db-serving-cert\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.165812 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.165794    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70b55c16-655f-41b4-aea8-ea598a609543-cert\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.177868 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.177813    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vts5h\" (UniqueName: \"kubernetes.io/projected/703b350d-5221-4434-b101-fe395ca946db-kube-api-access-vts5h\") pod \"console-operator-68b757fb67-ctnnf\" (UID: \"703b350d-5221-4434-b101-fe395ca946db\") " pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.178595 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.178182    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfqn\" (UniqueName: \"kubernetes.io/projected/70b55c16-655f-41b4-aea8-ea598a609543-kube-api-access-fsfqn\") pod \"ingress-canary-hjr72\" (UID: \"70b55c16-655f-41b4-aea8-ea598a609543\") " pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.178595 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.178539    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:01.178760 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.178657    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk8gl\" (UniqueName: \"kubernetes.io/projected/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-kube-api-access-zk8gl\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.178855 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.178837    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cttc\" (UniqueName: \"kubernetes.io/projected/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-kube-api-access-5cttc\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.238572 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.238546    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"
Jul 02 18:31:01.289211 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.289179    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:01.301795 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.301773    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hjr72"
Jul 02 18:31:01.566785 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.566749    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:01.566995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.566867    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:01.566995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.566910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.566995 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.566923    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:01.566995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.566934    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:01.566995 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.566941    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567021    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.56699721 +0000 UTC m=+38.896246833 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567044    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567075    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567087    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.567068537 +0000 UTC m=+38.896318151 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567109    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.567101474 +0000 UTC m=+38.896351075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:01.567263 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.567128    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.567113078 +0000 UTC m=+38.896362693 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:01.667560 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.667527    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:01.667737 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:01.667595    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:01.667737 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.667707    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:01.667737 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.667722    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:01.667898 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.667780    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.66776195 +0000 UTC m=+38.997011557 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:01.667898 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:01.667795    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:02.667789263 +0000 UTC m=+38.997038864 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:02.575275 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.575236    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.575281    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.575307    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.575357    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575435    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.57541522 +0000 UTC m=+40.904664845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575423    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575481    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575489    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575512    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.575500303 +0000 UTC m=+40.904749918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575550    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.575537181 +0000 UTC m=+40.904786783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575493    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:02.575918 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.575584    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.575578496 +0000 UTC m=+40.904828116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:02.578874 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.578844    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"]
Jul 02 18:31:02.593734 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.593715    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"]
Jul 02 18:31:02.593841 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.593815    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.596479 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.596447    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:02.596479 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.596458    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 02 18:31:02.596863 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.596789    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 02 18:31:02.596967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.596863    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 02 18:31:02.597925 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.597907    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-4hdjl\""
Jul 02 18:31:02.676620 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.676595    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe01e806-7bf3-4328-864c-e78edf501e2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.676761 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.676679    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:02.676761 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.676703    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe01e806-7bf3-4328-864c-e78edf501e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.676761 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.676730    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:02.676761 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.676747    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756f4\" (UniqueName: \"kubernetes.io/projected/fe01e806-7bf3-4328-864c-e78edf501e2f-kube-api-access-756f4\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.676934 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.676825    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:02.676934 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.676844    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:02.676934 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.676896    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.676879042 +0000 UTC m=+41.006128643 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:02.676934 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:02.676911    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:04.676904821 +0000 UTC m=+41.006154423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:02.777195 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.777161    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe01e806-7bf3-4328-864c-e78edf501e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.777341 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.777220    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756f4\" (UniqueName: \"kubernetes.io/projected/fe01e806-7bf3-4328-864c-e78edf501e2f-kube-api-access-756f4\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.777341 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.777276    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe01e806-7bf3-4328-864c-e78edf501e2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.777789 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.777773    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe01e806-7bf3-4328-864c-e78edf501e2f-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.780325 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.780268    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fe01e806-7bf3-4328-864c-e78edf501e2f-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.787010 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.786941    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756f4\" (UniqueName: \"kubernetes.io/projected/fe01e806-7bf3-4328-864c-e78edf501e2f-kube-api-access-756f4\") pod \"kube-storage-version-migrator-operator-846bf668f7-kh477\" (UID: \"fe01e806-7bf3-4328-864c-e78edf501e2f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.898563 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.898391    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-6h2w9"]
Jul 02 18:31:02.904186 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.904160    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"
Jul 02 18:31:02.905663 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.905526    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hjr72"]
Jul 02 18:31:02.906459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.906437    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp"]
Jul 02 18:31:02.907669 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:02.907650    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-ctnnf"]
Jul 02 18:31:02.974266 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:02.974242    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88ac6aaf_06fa_4909_ae0d_14b8849883b8.slice/crio-04df5c737b14e3fb04ebf95d57262bc82667363c507ed6d9fb6d0d15cf961012 WatchSource:0}: Error finding container 04df5c737b14e3fb04ebf95d57262bc82667363c507ed6d9fb6d0d15cf961012: Status 404 returned error can't find the container with id 04df5c737b14e3fb04ebf95d57262bc82667363c507ed6d9fb6d0d15cf961012
Jul 02 18:31:02.975097 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:02.975074    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0979127_7619_4133_a067_e6f9c0078edd.slice/crio-3d744241bbca7ecb4e88c289a1cee9628b1783b32a511e69a78150865d4a5c24 WatchSource:0}: Error finding container 3d744241bbca7ecb4e88c289a1cee9628b1783b32a511e69a78150865d4a5c24: Status 404 returned error can't find the container with id 3d744241bbca7ecb4e88c289a1cee9628b1783b32a511e69a78150865d4a5c24
Jul 02 18:31:02.976041 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:02.976007    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b55c16_655f_41b4_aea8_ea598a609543.slice/crio-e52f08700783e68343b89fbf569b57c5416f26eef412c2c4f292038d43f380eb WatchSource:0}: Error finding container e52f08700783e68343b89fbf569b57c5416f26eef412c2c4f292038d43f380eb: Status 404 returned error can't find the container with id e52f08700783e68343b89fbf569b57c5416f26eef412c2c4f292038d43f380eb
Jul 02 18:31:02.976521 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:02.976463    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703b350d_5221_4434_b101_fe395ca946db.slice/crio-206b5842e493a213a85424994431c3ad7bd1adcde8ee9f02451be1364d0d4b6a WatchSource:0}: Error finding container 206b5842e493a213a85424994431c3ad7bd1adcde8ee9f02451be1364d0d4b6a: Status 404 returned error can't find the container with id 206b5842e493a213a85424994431c3ad7bd1adcde8ee9f02451be1364d0d4b6a
Jul 02 18:31:03.140514 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.140446    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477"]
Jul 02 18:31:03.144642 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:03.144608    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe01e806_7bf3_4328_864c_e78edf501e2f.slice/crio-d9efeda99c0fa493383108feb25235b2a0b591dfd831f56387cb5f02a2545a43 WatchSource:0}: Error finding container d9efeda99c0fa493383108feb25235b2a0b591dfd831f56387cb5f02a2545a43: Status 404 returned error can't find the container with id d9efeda99c0fa493383108feb25235b2a0b591dfd831f56387cb5f02a2545a43
Jul 02 18:31:03.611403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.611361    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:03.675549 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.675513    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" event={"ID":"703b350d-5221-4434-b101-fe395ca946db","Type":"ContainerStarted","Data":"206b5842e493a213a85424994431c3ad7bd1adcde8ee9f02451be1364d0d4b6a"}
Jul 02 18:31:03.677013 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.676948    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp" event={"ID":"f0979127-7619-4133-a067-e6f9c0078edd","Type":"ContainerStarted","Data":"3d744241bbca7ecb4e88c289a1cee9628b1783b32a511e69a78150865d4a5c24"}
Jul 02 18:31:03.680461 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.680410    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjr72" event={"ID":"70b55c16-655f-41b4-aea8-ea598a609543","Type":"ContainerStarted","Data":"e52f08700783e68343b89fbf569b57c5416f26eef412c2c4f292038d43f380eb"}
Jul 02 18:31:03.685940 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.685906    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="219c21d8c9d2f9a022e4cea5c0bdcd8d98b65eaaf602e3c43ea424ed0a3caa97" exitCode=0
Jul 02 18:31:03.686055 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.686003    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"219c21d8c9d2f9a022e4cea5c0bdcd8d98b65eaaf602e3c43ea424ed0a3caa97"}
Jul 02 18:31:03.690964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.690939    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477" event={"ID":"fe01e806-7bf3-4328-864c-e78edf501e2f","Type":"ContainerStarted","Data":"d9efeda99c0fa493383108feb25235b2a0b591dfd831f56387cb5f02a2545a43"}
Jul 02 18:31:03.692474 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.692448    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9" event={"ID":"88ac6aaf-06fa-4909-ae0d-14b8849883b8","Type":"ContainerStarted","Data":"04df5c737b14e3fb04ebf95d57262bc82667363c507ed6d9fb6d0d15cf961012"}
Jul 02 18:31:03.716845 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:03.716725    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.580425 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.580092    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.592814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.592787    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:04.592931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.592838    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:04.592931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.592868    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:04.592931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.592923    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:04.593102 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593081    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:04.593102 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593096    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:04.593196 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593149    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.593129748 +0000 UTC m=+44.922379366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:04.593256 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593215    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:04.593256 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593254    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.593241826 +0000 UTC m=+44.922491442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:04.593357 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593319    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.593309652 +0000 UTC m=+44.922559270 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:04.593417 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593374    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:04.593417 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.593405    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.59339532 +0000 UTC m=+44.922644924 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:04.595267 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.594120    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"]
Jul 02 18:31:04.610804 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.610114    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"]
Jul 02 18:31:04.610804 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.610211    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.613836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.613467    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 18:31:04.613836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.615717    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-zm2dz\""
Jul 02 18:31:04.613836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.615925    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 18:31:04.616578 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.616238    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:04.616578 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.616400    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 18:31:04.655205 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.655154    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.693343    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06c2f72b-be90-44b8-9188-a5f8f32b297c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.693399    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06c2f72b-be90-44b8-9188-a5f8f32b297c-config\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.693511    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.693576    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.693624    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbb8r\" (UniqueName: \"kubernetes.io/projected/06c2f72b-be90-44b8-9188-a5f8f32b297c-kube-api-access-rbb8r\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.693804    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.693873    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.693853597 +0000 UTC m=+45.023103213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.694037    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:04.694241 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:04.694110    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:08.694090521 +0000 UTC m=+45.023340124 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:04.700199 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.700167    2553 generic.go:356] "Generic (PLEG): container finished" podID="9adbcc24-afe2-400e-8524-fad68fdc6797" containerID="26a5e9108ca137f1ae324e8b9ab630568e1bd3cfd940bb60261f44f968889963" exitCode=0
Jul 02 18:31:04.700330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.700271    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerDied","Data":"26a5e9108ca137f1ae324e8b9ab630568e1bd3cfd940bb60261f44f968889963"}
Jul 02 18:31:04.702908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.702875    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp" event={"ID":"f0979127-7619-4133-a067-e6f9c0078edd","Type":"ContainerStarted","Data":"75d1005241619f60f91281f1dbdd81afb1f471838d6ddfee3cec0f2e734b71a0"}
Jul 02 18:31:04.732675 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.732643    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.742902 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.742846    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-ftjdp" podStartSLOduration=5.265591334 podStartE2EDuration="6.742824724s" totalImagesPullingTime="1.47723339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:04.741568026 +0000 UTC m=+41.070817650" watchObservedRunningTime="2026-07-02 18:31:04.742824724 +0000 UTC m=+41.072074348"
Jul 02 18:31:04.795072 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.795030    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06c2f72b-be90-44b8-9188-a5f8f32b297c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.795217 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.795188    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06c2f72b-be90-44b8-9188-a5f8f32b297c-config\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.795470 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.795450    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbb8r\" (UniqueName: \"kubernetes.io/projected/06c2f72b-be90-44b8-9188-a5f8f32b297c-kube-api-access-rbb8r\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.796422 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.796333    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/06c2f72b-be90-44b8-9188-a5f8f32b297c-config\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.800846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.800726    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/06c2f72b-be90-44b8-9188-a5f8f32b297c-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.807649 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.806401    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbb8r\" (UniqueName: \"kubernetes.io/projected/06c2f72b-be90-44b8-9188-a5f8f32b297c-kube-api-access-rbb8r\") pod \"service-ca-operator-5d46d9ffd4-z669j\" (UID: \"06c2f72b-be90-44b8-9188-a5f8f32b297c\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.852668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.852610    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:04.928385 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.928156    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"
Jul 02 18:31:04.952995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:04.952944    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:05.709693 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:05.709648    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vmncx" event={"ID":"9adbcc24-afe2-400e-8524-fad68fdc6797","Type":"ContainerStarted","Data":"d5924b7ef3d25cd9056b3b5af6452bae7cbbb180287842a0d43fc9284ecfc4f6"}
Jul 02 18:31:05.733131 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:05.733073    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vmncx" podStartSLOduration=9.596232074 podStartE2EDuration="40.733052484s" totalImagesPullingTime="31.13682041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:05.732455409 +0000 UTC m=+42.061705034" watchObservedRunningTime="2026-07-02 18:31:05.733052484 +0000 UTC m=+42.062302109"
Jul 02 18:31:05.852823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:05.852782    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:05.915206 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:05.915173    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.162271 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.162245    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"]
Jul 02 18:31:06.180896 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.180866    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"]
Jul 02 18:31:06.181079 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.180997    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"
Jul 02 18:31:06.183551 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.183532    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rj2s8\""
Jul 02 18:31:06.309888 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.309854    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pft7x\" (UniqueName: \"kubernetes.io/projected/1986e221-4161-42ef-a43a-6fbef1f85c5e-kube-api-access-pft7x\") pod \"network-check-source-989f8b8dc-drlq7\" (UID: \"1986e221-4161-42ef-a43a-6fbef1f85c5e\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"
Jul 02 18:31:06.411131 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.411090    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pft7x\" (UniqueName: \"kubernetes.io/projected/1986e221-4161-42ef-a43a-6fbef1f85c5e-kube-api-access-pft7x\") pod \"network-check-source-989f8b8dc-drlq7\" (UID: \"1986e221-4161-42ef-a43a-6fbef1f85c5e\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"
Jul 02 18:31:06.432015 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.431922    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pft7x\" (UniqueName: \"kubernetes.io/projected/1986e221-4161-42ef-a43a-6fbef1f85c5e-kube-api-access-pft7x\") pod \"network-check-source-989f8b8dc-drlq7\" (UID: \"1986e221-4161-42ef-a43a-6fbef1f85c5e\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"
Jul 02 18:31:06.451515 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.451478    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.491202 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.491163    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"
Jul 02 18:31:06.551544 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.551507    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:06.652949 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:06.652912    2553 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log" limit=1024
Jul 02 18:31:08.164316 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.164277    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7"]
Jul 02 18:31:08.171419 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:08.171242    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1986e221_4161_42ef_a43a_6fbef1f85c5e.slice/crio-9cdfccdacb5626015bf14a76404ef8625a5edac060586ea34ce5ae10d132afb1 WatchSource:0}: Error finding container 9cdfccdacb5626015bf14a76404ef8625a5edac060586ea34ce5ae10d132afb1: Status 404 returned error can't find the container with id 9cdfccdacb5626015bf14a76404ef8625a5edac060586ea34ce5ae10d132afb1
Jul 02 18:31:08.181625 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.181600    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j"]
Jul 02 18:31:08.191067 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:08.191037    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c2f72b_be90_44b8_9188_a5f8f32b297c.slice/crio-81ef6358adfb47dba5beca0fe6f8e27e904ec4e4ea57320ac5d2d1d85d98e464 WatchSource:0}: Error finding container 81ef6358adfb47dba5beca0fe6f8e27e904ec4e4ea57320ac5d2d1d85d98e464: Status 404 returned error can't find the container with id 81ef6358adfb47dba5beca0fe6f8e27e904ec4e4ea57320ac5d2d1d85d98e464
Jul 02 18:31:08.631411 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.631355    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:08.631608 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.631500    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:08.631608 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.631543    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:08.631608 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631546    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:08.631608 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631568    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:08.631608 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.631571    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631634    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.631613947 +0000 UTC m=+52.960863568 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631667    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.631656576 +0000 UTC m=+52.960906183 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631672    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631689    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631772    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.631753352 +0000 UTC m=+52.961002960 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:08.631867 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.631795    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.631783453 +0000 UTC m=+52.961033059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:08.727330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.727239    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hjr72" event={"ID":"70b55c16-655f-41b4-aea8-ea598a609543","Type":"ContainerStarted","Data":"94c71de3959a3419a4db93b3c44072e2458e864f9713684e7c5e5f62f5915569"}
Jul 02 18:31:08.729950 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.729872    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477" event={"ID":"fe01e806-7bf3-4328-864c-e78edf501e2f","Type":"ContainerStarted","Data":"056cf11aeaeaef51e944e1fe6088d3036a58d8a8123176f87e55b6d5a502cdbe"}
Jul 02 18:31:08.732166 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.732128    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:08.732316 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.732191    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:08.732388 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.732365    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9" event={"ID":"88ac6aaf-06fa-4909-ae0d-14b8849883b8","Type":"ContainerStarted","Data":"454a6e64c915a367dfe7661b367180ba351ec63623146feb5050c29a0f51bb8e"}
Jul 02 18:31:08.732445 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.732387    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:08.732445 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.732441    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.732424072 +0000 UTC m=+53.061673691 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:08.732787 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.732768    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:08.733003 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:08.732819    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:16.732805838 +0000 UTC m=+53.062055446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:08.734766 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.734705    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7" event={"ID":"1986e221-4161-42ef-a43a-6fbef1f85c5e","Type":"ContainerStarted","Data":"9cdfccdacb5626015bf14a76404ef8625a5edac060586ea34ce5ae10d132afb1"}
Jul 02 18:31:08.736744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.736727    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/0.log"
Jul 02 18:31:08.736845 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.736764    2553 generic.go:356] "Generic (PLEG): container finished" podID="703b350d-5221-4434-b101-fe395ca946db" containerID="4bc8b3d2c7326e0a6c0664d62bebfa172d48d3f38f8604d67c1c7c5d2e63e6f3" exitCode=255
Jul 02 18:31:08.736911 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.736850    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" event={"ID":"703b350d-5221-4434-b101-fe395ca946db","Type":"ContainerDied","Data":"4bc8b3d2c7326e0a6c0664d62bebfa172d48d3f38f8604d67c1c7c5d2e63e6f3"}
Jul 02 18:31:08.737039 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.737021    2553 scope.go:117] "RemoveContainer" containerID="4bc8b3d2c7326e0a6c0664d62bebfa172d48d3f38f8604d67c1c7c5d2e63e6f3"
Jul 02 18:31:08.739617 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.739593    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j" event={"ID":"06c2f72b-be90-44b8-9188-a5f8f32b297c","Type":"ContainerStarted","Data":"81ef6358adfb47dba5beca0fe6f8e27e904ec4e4ea57320ac5d2d1d85d98e464"}
Jul 02 18:31:08.745310 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.745262    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hjr72" podStartSLOduration=3.7052896730000002 podStartE2EDuration="8.745244502s" totalImagesPullingTime="5.039954829s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:08.744622501 +0000 UTC m=+45.073872125" watchObservedRunningTime="2026-07-02 18:31:08.745244502 +0000 UTC m=+45.074494127"
Jul 02 18:31:08.784557 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.783800    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477" podStartSLOduration=1.9096046260000001 podStartE2EDuration="6.783777366s" totalImagesPullingTime="4.87417274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:08.782598523 +0000 UTC m=+45.111848147" watchObservedRunningTime="2026-07-02 18:31:08.783777366 +0000 UTC m=+45.113026991"
Jul 02 18:31:08.807049 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:08.806989    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9" podStartSLOduration=5.763480321 podStartE2EDuration="10.806951297s" totalImagesPullingTime="5.043470976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:08.806399194 +0000 UTC m=+45.135648812" watchObservedRunningTime="2026-07-02 18:31:08.806951297 +0000 UTC m=+45.136200922"
Jul 02 18:31:09.746585 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.746555    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/1.log"
Jul 02 18:31:09.747053 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.747033    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/0.log"
Jul 02 18:31:09.747131 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.747075    2553 generic.go:356] "Generic (PLEG): container finished" podID="703b350d-5221-4434-b101-fe395ca946db" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530" exitCode=255
Jul 02 18:31:09.747249 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.747226    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" event={"ID":"703b350d-5221-4434-b101-fe395ca946db","Type":"ContainerDied","Data":"42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"}
Jul 02 18:31:09.747316 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.747304    2553 scope.go:117] "RemoveContainer" containerID="4bc8b3d2c7326e0a6c0664d62bebfa172d48d3f38f8604d67c1c7c5d2e63e6f3"
Jul 02 18:31:09.747655 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.747616    2553 scope.go:117] "RemoveContainer" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"
Jul 02 18:31:09.747865 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:09.747838    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:09.895717 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.895675    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"]
Jul 02 18:31:09.928487 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.926934    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"]
Jul 02 18:31:09.928487 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.927115    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"
Jul 02 18:31:09.929857 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.929834    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-5wjx7\""
Jul 02 18:31:09.930406 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.930385    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 18:31:09.930528 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.930407    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 18:31:09.947194 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.946292    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2zr"]
Jul 02 18:31:09.964641 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.964601    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:09.967788 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.967760    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2zr"]
Jul 02 18:31:09.968318 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.968272    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 18:31:09.968694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.968654    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 18:31:09.969486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:09.969462    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5tcqj\""
Jul 02 18:31:10.052120 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052024    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a844e35-052f-48ba-bb97-675804539124-data-volume\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.052120 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052086    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fwhd\" (UniqueName: \"kubernetes.io/projected/a1be429c-7b16-4375-ac8d-d5a97f8dfb54-kube-api-access-4fwhd\") pod \"migrator-d57d58cc9-fqslg\" (UID: \"a1be429c-7b16-4375-ac8d-d5a97f8dfb54\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"
Jul 02 18:31:10.052361 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052124    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.052361 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052300    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m5sj\" (UniqueName: \"kubernetes.io/projected/9a844e35-052f-48ba-bb97-675804539124-kube-api-access-4m5sj\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.052460 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052376    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a844e35-052f-48ba-bb97-675804539124-crio-socket\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.052460 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.052404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.144390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.144350    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"]
Jul 02 18:31:10.153205 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153163    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m5sj\" (UniqueName: \"kubernetes.io/projected/9a844e35-052f-48ba-bb97-675804539124-kube-api-access-4m5sj\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.153374 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153251    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a844e35-052f-48ba-bb97-675804539124-crio-socket\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.153374 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153275    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.153374 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153363    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a844e35-052f-48ba-bb97-675804539124-data-volume\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.153567 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153390    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fwhd\" (UniqueName: \"kubernetes.io/projected/a1be429c-7b16-4375-ac8d-d5a97f8dfb54-kube-api-access-4fwhd\") pod \"migrator-d57d58cc9-fqslg\" (UID: \"a1be429c-7b16-4375-ac8d-d5a97f8dfb54\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"
Jul 02 18:31:10.153567 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.153418    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.153672 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.153615    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:31:10.153727 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.153681    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:10.653661129 +0000 UTC m=+46.982910731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : configmap "kube-rbac-proxy" not found
Jul 02 18:31:10.154543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.154245    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a844e35-052f-48ba-bb97-675804539124-crio-socket\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.154543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.154327    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a844e35-052f-48ba-bb97-675804539124-data-volume\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.154543 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.154428    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:10.154543 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.154474    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:10.654459459 +0000 UTC m=+46.983709063 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:10.171292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.171227    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m5sj\" (UniqueName: \"kubernetes.io/projected/9a844e35-052f-48ba-bb97-675804539124-kube-api-access-4m5sj\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.171292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.171272    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fwhd\" (UniqueName: \"kubernetes.io/projected/a1be429c-7b16-4375-ac8d-d5a97f8dfb54-kube-api-access-4fwhd\") pod \"migrator-d57d58cc9-fqslg\" (UID: \"a1be429c-7b16-4375-ac8d-d5a97f8dfb54\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"
Jul 02 18:31:10.183181 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.183148    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.189364 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.189331    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 18:31:10.192995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.189598    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 18:31:10.192995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.189661    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-km8vj\""
Jul 02 18:31:10.192995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.191662    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"]
Jul 02 18:31:10.241302 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.241253    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"
Jul 02 18:31:10.254856 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.254807    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.255117 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.255081    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.356256 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.356164    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.356420 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.356320    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.356470 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.356449    2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 18:31:10.356516 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.356504    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert podName:1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:10.856489649 +0000 UTC m=+47.185739250 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2ss9" (UID: "1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20") : secret "networking-console-plugin-cert" not found
Jul 02 18:31:10.356944 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.356908    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.659067 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.658959    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.659212 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.659071    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:31:10.659212 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.659141    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:11.659124025 +0000 UTC m=+47.988373626 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : configmap "kube-rbac-proxy" not found
Jul 02 18:31:10.659212 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.659190    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:10.659372 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.659354    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:10.659429 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.659417    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:11.659398755 +0000 UTC m=+47.988648376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:10.752340 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.752310    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/1.log"
Jul 02 18:31:10.752824 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.752709    2553 scope.go:117] "RemoveContainer" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"
Jul 02 18:31:10.752952 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.752919    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:10.861599 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:10.861562    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:10.861793 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.861737    2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 18:31:10.861860 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:10.861812    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert podName:1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:11.8617891 +0000 UTC m=+48.191038719 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2ss9" (UID: "1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20") : secret "networking-console-plugin-cert" not found
Jul 02 18:31:11.133661 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.133622    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxjbg_a3da5140-4064-4577-8191-4667846a5726/dns-node-resolver/0.log"
Jul 02 18:31:11.289717 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.289669    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:11.289894 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.289751    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:11.669752 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.669716    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:11.669911 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.669778    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:11.669911 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.669860    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:11.669911 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.669901    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:31:11.670045 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.669923    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.669908413 +0000 UTC m=+49.999158014 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:11.670045 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.669947    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.66993492 +0000 UTC m=+49.999184521 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : configmap "kube-rbac-proxy" not found
Jul 02 18:31:11.756380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.756351    2553 scope.go:117] "RemoveContainer" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"
Jul 02 18:31:11.756735 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.756518    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:11.871125 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.871091    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:11.871277 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.871209    2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 18:31:11.871321 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:11.871281    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert podName:1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:13.871260912 +0000 UTC m=+50.200510519 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2ss9" (UID: "1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20") : secret "networking-console-plugin-cert" not found
Jul 02 18:31:11.933503 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:11.933447    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27jxq_edc54bf1-8835-4198-91aa-4a09c593c2df/node-ca/0.log"
Jul 02 18:31:12.167176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.167147    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg"]
Jul 02 18:31:12.169991 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:12.169953    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1be429c_7b16_4375_ac8d_d5a97f8dfb54.slice/crio-77afaa27993a15a455252cefc2f0548a4ab12b83fa6b44a65333ce7fb6a2fb06 WatchSource:0}: Error finding container 77afaa27993a15a455252cefc2f0548a4ab12b83fa6b44a65333ce7fb6a2fb06: Status 404 returned error can't find the container with id 77afaa27993a15a455252cefc2f0548a4ab12b83fa6b44a65333ce7fb6a2fb06
Jul 02 18:31:12.733310 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.733279    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjr72_70b55c16-655f-41b4-aea8-ea598a609543/serve-healthcheck-canary/0.log"
Jul 02 18:31:12.761423 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.760713    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j" event={"ID":"06c2f72b-be90-44b8-9188-a5f8f32b297c","Type":"ContainerStarted","Data":"7763aecc7d3084c9e34b0dcfd62338b19d6debe202a1cde2fc1decfba30ebb70"}
Jul 02 18:31:12.763382 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.763352    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7" event={"ID":"1986e221-4161-42ef-a43a-6fbef1f85c5e","Type":"ContainerStarted","Data":"a6e57e5140063a80c5a7435d9d8b1dac5f302be062f63bdb4cce92388a034484"}
Jul 02 18:31:12.764776 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.764750    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg" event={"ID":"a1be429c-7b16-4375-ac8d-d5a97f8dfb54","Type":"ContainerStarted","Data":"77afaa27993a15a455252cefc2f0548a4ab12b83fa6b44a65333ce7fb6a2fb06"}
Jul 02 18:31:12.779011 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.778902    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j" podStartSLOduration=4.925625185 podStartE2EDuration="8.778888484s" totalImagesPullingTime="3.853263299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:12.777887043 +0000 UTC m=+49.107136667" watchObservedRunningTime="2026-07-02 18:31:12.778888484 +0000 UTC m=+49.108138116"
Jul 02 18:31:12.799752 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:12.799701    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-drlq7" podStartSLOduration=2.916645254 podStartE2EDuration="6.799682216s" totalImagesPullingTime="3.883036962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:12.798942129 +0000 UTC m=+49.128191756" watchObservedRunningTime="2026-07-02 18:31:12.799682216 +0000 UTC m=+49.128931832"
Jul 02 18:31:13.689019 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:13.688991    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:13.689130 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:13.689054    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:13.689178 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.689154    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:13.689232 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.689185    2553 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 18:31:13.689232 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.689224    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:17.689205493 +0000 UTC m=+54.018455096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:13.689382 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.689240    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:17.689233401 +0000 UTC m=+54.018483002 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : configmap "kube-rbac-proxy" not found
Jul 02 18:31:13.769635 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:13.769601    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg" event={"ID":"a1be429c-7b16-4375-ac8d-d5a97f8dfb54","Type":"ContainerStarted","Data":"e695318a36517f91d205ce7ef0a6aaf0e580b2dd7361da8cb39c28b2f5d03f9d"}
Jul 02 18:31:13.890544 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:13.890472    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:13.890675 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.890624    2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 18:31:13.890730 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:13.890692    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert podName:1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:17.89067201 +0000 UTC m=+54.219921629 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2ss9" (UID: "1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20") : secret "networking-console-plugin-cert" not found
Jul 02 18:31:14.777190 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:14.777158    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg" event={"ID":"a1be429c-7b16-4375-ac8d-d5a97f8dfb54","Type":"ContainerStarted","Data":"94cdc615112b104145b0a891175a08488f77a9ab9e52010b73cccc9c30cc28d4"}
Jul 02 18:31:14.792967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:14.792916    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-fqslg" podStartSLOduration=4.329812246 podStartE2EDuration="5.792900514s" totalImagesPullingTime="1.463088268s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:14.792556385 +0000 UTC m=+51.121806008" watchObservedRunningTime="2026-07-02 18:31:14.792900514 +0000 UTC m=+51.122150137"
Jul 02 18:31:15.718337 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.718299    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-c6c5c"]
Jul 02 18:31:15.732747 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.732713    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-c6c5c"]
Jul 02 18:31:15.732892 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.732842    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.735554 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.735532    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 02 18:31:15.735818 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.735799    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 02 18:31:15.736871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.736856    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-5wfhl\""
Jul 02 18:31:15.736961 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.736918    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 02 18:31:15.736961 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.736930    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 02 18:31:15.809064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.809030    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvtw8\" (UniqueName: \"kubernetes.io/projected/b6cdad12-0301-4134-9f2e-17c769d711ba-kube-api-access-zvtw8\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.809414 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.809092    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-key\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.809414 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.809302    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-cabundle\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.910021 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.909964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-cabundle\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.910189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.910058    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvtw8\" (UniqueName: \"kubernetes.io/projected/b6cdad12-0301-4134-9f2e-17c769d711ba-kube-api-access-zvtw8\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.910189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.910121    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-key\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.910807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.910772    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-cabundle\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.915605 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.915569    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b6cdad12-0301-4134-9f2e-17c769d711ba-signing-key\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:15.926192 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:15.926160    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvtw8\" (UniqueName: \"kubernetes.io/projected/b6cdad12-0301-4134-9f2e-17c769d711ba-kube-api-access-zvtw8\") pod \"service-ca-758f599c5-c6c5c\" (UID: \"b6cdad12-0301-4134-9f2e-17c769d711ba\") " pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:16.049958 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.049886    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-c6c5c"
Jul 02 18:31:16.180437 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.180407    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-c6c5c"]
Jul 02 18:31:16.183150 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:16.183120    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6cdad12_0301_4134_9f2e_17c769d711ba.slice/crio-1b83a1e950300f4b3635afeda0ce2e403755108d259f28cc875893d206dbec39 WatchSource:0}: Error finding container 1b83a1e950300f4b3635afeda0ce2e403755108d259f28cc875893d206dbec39: Status 404 returned error can't find the container with id 1b83a1e950300f4b3635afeda0ce2e403755108d259f28cc875893d206dbec39
Jul 02 18:31:16.717216 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.717181    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:16.717216 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.717226    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:16.717407 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.717257    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:16.717407 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.717311    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:16.717407 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717353    2553 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717402    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.717377033 +0000 UTC m=+69.046626638 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : configmap references non-existent config key: service-ca.crt
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717419    2553 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717429    2553 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7cddf47b9b-ggzk6: secret "image-registry-tls" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717430    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls podName:f0b5acde-5fb9-4fbc-8a51-589ec242aa45 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.717420679 +0000 UTC m=+69.046670287 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-lhx8q" (UID: "f0b5acde-5fb9-4fbc-8a51-589ec242aa45") : secret "cluster-monitoring-operator-tls" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717410    2553 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717460    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls podName:05eea021-83b5-4e47-b117-154bb3b30d32 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.717446633 +0000 UTC m=+69.046696236 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls") pod "image-registry-7cddf47b9b-ggzk6" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32") : secret "image-registry-tls" not found
Jul 02 18:31:16.717563 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.717478    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs podName:97f2cfcc-f466-4c0f-a731-0adf39a75625 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.717468582 +0000 UTC m=+69.046718183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs") pod "router-default-674c7d54dc-s8b5k" (UID: "97f2cfcc-f466-4c0f-a731-0adf39a75625") : secret "router-metrics-certs-default" not found
Jul 02 18:31:16.785895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.785859    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-c6c5c" event={"ID":"b6cdad12-0301-4134-9f2e-17c769d711ba","Type":"ContainerStarted","Data":"278b3b713f734d9af7e0ada422f755536c9853f79eb179cedf2ecf33bbc0c086"}
Jul 02 18:31:16.786058 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.785911    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-c6c5c" event={"ID":"b6cdad12-0301-4134-9f2e-17c769d711ba","Type":"ContainerStarted","Data":"1b83a1e950300f4b3635afeda0ce2e403755108d259f28cc875893d206dbec39"}
Jul 02 18:31:16.804490 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.804445    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-c6c5c" podStartSLOduration=1.804431325 podStartE2EDuration="1.804431325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:16.803890735 +0000 UTC m=+53.133140359" watchObservedRunningTime="2026-07-02 18:31:16.804431325 +0000 UTC m=+53.133680947"
Jul 02 18:31:16.817781 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.817751    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:16.818161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:16.817817    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:16.818161 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.817900    2553 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 18:31:16.818161 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.817949    2553 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 18:31:16.818161 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.817962    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls podName:93a17f0a-0ba7-45e1-ae42-f87237ee3e7e nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.81794814 +0000 UTC m=+69.147197741 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls") pod "dns-default-5wbwr" (UID: "93a17f0a-0ba7-45e1-ae42-f87237ee3e7e") : secret "dns-default-metrics-tls" not found
Jul 02 18:31:16.818161 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:16.818015    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls podName:b21a829d-ec29-4eaf-9f02-5ab7fdb71643 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:32.818003841 +0000 UTC m=+69.147253465 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-b7q45" (UID: "b21a829d-ec29-4eaf-9f02-5ab7fdb71643") : secret "samples-operator-tls" not found
Jul 02 18:31:17.726793 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:17.726740    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:17.727214 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:17.727031    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:17.727214 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:17.727187    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:17.727327 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:17.727251    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:25.727232696 +0000 UTC m=+62.056482304 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:17.727471 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:17.727448    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a844e35-052f-48ba-bb97-675804539124-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:17.929448 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:17.929410    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:17.929930 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:17.929570    2553 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 02 18:31:17.929930 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:17.929633    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert podName:1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:25.929614796 +0000 UTC m=+62.258864401 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-b2ss9" (UID: "1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20") : secret "networking-console-plugin-cert" not found
Jul 02 18:31:25.803962 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:25.803912    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:25.804379 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:25.804084    2553 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:25.804379 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:25.804153    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls podName:9a844e35-052f-48ba-bb97-675804539124 nodeName:}" failed. No retries permitted until 2026-07-02 18:31:41.804137519 +0000 UTC m=+78.133387119 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zh2zr" (UID: "9a844e35-052f-48ba-bb97-675804539124") : secret "insights-runtime-extractor-tls" not found
Jul 02 18:31:26.006169 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.006133    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:26.008670 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.008633    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-b2ss9\" (UID: \"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:26.102939 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.102846    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-km8vj\""
Jul 02 18:31:26.111175 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.111153    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"
Jul 02 18:31:26.157165 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.157132    2553 scope.go:117] "RemoveContainer" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"
Jul 02 18:31:26.252740 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.252656    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9"]
Jul 02 18:31:26.254962 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:26.254933    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a3ae2c5_aebc_4d0b_873c_b9ef73bc8c20.slice/crio-ce02c3453154f8642ed098dd98aa1856516f445f905a9ce83f5a582847651879 WatchSource:0}: Error finding container ce02c3453154f8642ed098dd98aa1856516f445f905a9ce83f5a582847651879: Status 404 returned error can't find the container with id ce02c3453154f8642ed098dd98aa1856516f445f905a9ce83f5a582847651879
Jul 02 18:31:26.819682 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.819640    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:31:26.820200 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.820181    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/1.log"
Jul 02 18:31:26.820257 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.820219    2553 generic.go:356] "Generic (PLEG): container finished" podID="703b350d-5221-4434-b101-fe395ca946db" containerID="688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72" exitCode=255
Jul 02 18:31:26.820330 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.820312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" event={"ID":"703b350d-5221-4434-b101-fe395ca946db","Type":"ContainerDied","Data":"688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72"}
Jul 02 18:31:26.820385 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.820367    2553 scope.go:117] "RemoveContainer" containerID="42e22c7c498ab4b968c9f877116758743b1fa52957067cf85586aeb8e0f66530"
Jul 02 18:31:26.820807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.820773    2553 scope.go:117] "RemoveContainer" containerID="688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72"
Jul 02 18:31:26.820986 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:26.820951    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:26.821866 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:26.821835    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9" event={"ID":"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20","Type":"ContainerStarted","Data":"ce02c3453154f8642ed098dd98aa1856516f445f905a9ce83f5a582847651879"}
Jul 02 18:31:27.666589 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:27.666567    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n2nk2"
Jul 02 18:31:27.826516 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:27.826476    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9" event={"ID":"1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20","Type":"ContainerStarted","Data":"320cee11fb24a7c022034fad6d1f5701e382aaba2d8da7bed647bc81b0c8a41f"}
Jul 02 18:31:27.828026 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:27.828001    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:31:27.842106 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:27.842055    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-b2ss9" podStartSLOduration=16.878989584 podStartE2EDuration="17.842038856s" totalImagesPullingTime="963.049272ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:27.841352884 +0000 UTC m=+64.170602508" watchObservedRunningTime="2026-07-02 18:31:27.842038856 +0000 UTC m=+64.171288493"
Jul 02 18:31:30.245052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.245004    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:31:30.245451 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.245171    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:31:30.248020 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.248004    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 18:31:30.249120 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.249105    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 18:31:30.258302 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.258281    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/86a22131-0126-4643-a945-41cbd09417dc-metrics-certs\") pod \"network-metrics-daemon-48vdd\" (UID: \"86a22131-0126-4643-a945-41cbd09417dc\") " pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:31:30.258302 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.258294    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/07a2791d-f19a-4d3a-9a53-1ec166771a43-original-pull-secret\") pod \"global-pull-secret-syncer-d5rdt\" (UID: \"07a2791d-f19a-4d3a-9a53-1ec166771a43\") " pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:31:30.272443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.272417    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-d5rdt"
Jul 02 18:31:30.294878 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.294853    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-86j9v\""
Jul 02 18:31:30.303043 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.303018    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-48vdd"
Jul 02 18:31:30.346588 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.346385    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:31:30.349383 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.349361    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l76f\" (UniqueName: \"kubernetes.io/projected/61507973-7cb0-44d4-850c-05b89d4c27fc-kube-api-access-7l76f\") pod \"network-check-target-kvrxf\" (UID: \"61507973-7cb0-44d4-850c-05b89d4c27fc\") " pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:31:30.406463 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:30.406419    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a2791d_f19a_4d3a_9a53_1ec166771a43.slice/crio-d5d4e99c48a75f0a7ac51d4224c01d00ba061cf95472d86e1fb5475fc10e14d8 WatchSource:0}: Error finding container d5d4e99c48a75f0a7ac51d4224c01d00ba061cf95472d86e1fb5475fc10e14d8: Status 404 returned error can't find the container with id d5d4e99c48a75f0a7ac51d4224c01d00ba061cf95472d86e1fb5475fc10e14d8
Jul 02 18:31:30.407880 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.407447    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-d5rdt"]
Jul 02 18:31:30.438914 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.438864    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-48vdd"]
Jul 02 18:31:30.441501 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:30.441478    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a22131_0126_4643_a945_41cbd09417dc.slice/crio-32ab7ba3e7ee59329fcf0b7fae1f3faa37d25a6832de1ed489519e08e39a9869 WatchSource:0}: Error finding container 32ab7ba3e7ee59329fcf0b7fae1f3faa37d25a6832de1ed489519e08e39a9869: Status 404 returned error can't find the container with id 32ab7ba3e7ee59329fcf0b7fae1f3faa37d25a6832de1ed489519e08e39a9869
Jul 02 18:31:30.589863 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.589778    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-vvptw\""
Jul 02 18:31:30.596914 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.596893    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:31:30.743628 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.743599    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvrxf"]
Jul 02 18:31:30.746318 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:30.746292    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61507973_7cb0_44d4_850c_05b89d4c27fc.slice/crio-6c90c2f183bab11b9ad632941258b054e65c86a10585ddbcf52f220bfcb94e06 WatchSource:0}: Error finding container 6c90c2f183bab11b9ad632941258b054e65c86a10585ddbcf52f220bfcb94e06: Status 404 returned error can't find the container with id 6c90c2f183bab11b9ad632941258b054e65c86a10585ddbcf52f220bfcb94e06
Jul 02 18:31:30.840477 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.840382    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d5rdt" event={"ID":"07a2791d-f19a-4d3a-9a53-1ec166771a43","Type":"ContainerStarted","Data":"d5d4e99c48a75f0a7ac51d4224c01d00ba061cf95472d86e1fb5475fc10e14d8"}
Jul 02 18:31:30.841850 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.841819    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48vdd" event={"ID":"86a22131-0126-4643-a945-41cbd09417dc","Type":"ContainerStarted","Data":"32ab7ba3e7ee59329fcf0b7fae1f3faa37d25a6832de1ed489519e08e39a9869"}
Jul 02 18:31:30.843590 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.843536    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvrxf" event={"ID":"61507973-7cb0-44d4-850c-05b89d4c27fc","Type":"ContainerStarted","Data":"78733f1be4ea1b82a65a743fb0360914380bdf923098aebd2faf8a11fcaa4460"}
Jul 02 18:31:30.843590 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.843586    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvrxf" event={"ID":"61507973-7cb0-44d4-850c-05b89d4c27fc","Type":"ContainerStarted","Data":"6c90c2f183bab11b9ad632941258b054e65c86a10585ddbcf52f220bfcb94e06"}
Jul 02 18:31:30.843786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.843668    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:31:30.863846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:30.863781    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kvrxf" podStartSLOduration=65.86375967 podStartE2EDuration="1m5.86375967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:30.862797415 +0000 UTC m=+67.192047037" watchObservedRunningTime="2026-07-02 18:31:30.86375967 +0000 UTC m=+67.193009309"
Jul 02 18:31:31.290178 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.290139    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:31.290641 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.290207    2553 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:31.290641 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.290620    2553 scope.go:117] "RemoveContainer" containerID="688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72"
Jul 02 18:31:31.290851 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:31.290827    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:31.849901 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.849809    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48vdd" event={"ID":"86a22131-0126-4643-a945-41cbd09417dc","Type":"ContainerStarted","Data":"ea7dcd4336ced03c3a02454c4e41508630385ae483249c875c2051611f02984a"}
Jul 02 18:31:31.850099 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.849914    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-48vdd" event={"ID":"86a22131-0126-4643-a945-41cbd09417dc","Type":"ContainerStarted","Data":"31d0765d216475d9d270db6757a0b6cc7d646d8ad7712fccb333ac2b6f8e33ff"}
Jul 02 18:31:31.869646 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:31.869584    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-48vdd" podStartSLOduration=65.907590762 podStartE2EDuration="1m6.869564294s" totalImagesPullingTime="961.973532ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:31.867421913 +0000 UTC m=+68.196671536" watchObservedRunningTime="2026-07-02 18:31:31.869564294 +0000 UTC m=+68.198813917"
Jul 02 18:31:32.774201 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.774167    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:32.774662 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.774234    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:32.774662 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.774273    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:32.774662 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.774331    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:32.774966 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.774939    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/97f2cfcc-f466-4c0f-a731-0adf39a75625-service-ca-bundle\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:32.777020 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.776992    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"image-registry-7cddf47b9b-ggzk6\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") " pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:32.777137 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.777027    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/f0b5acde-5fb9-4fbc-8a51-589ec242aa45-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-lhx8q\" (UID: \"f0b5acde-5fb9-4fbc-8a51-589ec242aa45\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:32.777137 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.777091    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/97f2cfcc-f466-4c0f-a731-0adf39a75625-metrics-certs\") pod \"router-default-674c7d54dc-s8b5k\" (UID: \"97f2cfcc-f466-4c0f-a731-0adf39a75625\") " pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:32.875720 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.875678    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:32.875903 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.875739    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:32.878401 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.878369    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/93a17f0a-0ba7-45e1-ae42-f87237ee3e7e-metrics-tls\") pod \"dns-default-5wbwr\" (UID: \"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e\") " pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:32.878533 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.878520    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/b21a829d-ec29-4eaf-9f02-5ab7fdb71643-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-b7q45\" (UID: \"b21a829d-ec29-4eaf-9f02-5ab7fdb71643\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:32.976359 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.976324    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-29f52\""
Jul 02 18:31:32.983900 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:32.983856    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:33.000720 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.000688    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-r2j6b\""
Jul 02 18:31:33.007988 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.007941    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:33.018680 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.018652    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-wxn7t\""
Jul 02 18:31:33.026981 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.026883    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"
Jul 02 18:31:33.063988 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.063944    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-wtqjs\""
Jul 02 18:31:33.072242 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.072205    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"
Jul 02 18:31:33.123300 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.123267    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xnjsw\""
Jul 02 18:31:33.130797 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:33.130771    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:34.174809 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.174719    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-674c7d54dc-s8b5k"]
Jul 02 18:31:34.184600 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:34.184561    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f2cfcc_f466_4c0f_a731_0adf39a75625.slice/crio-eb301b9ad8c54454478590a307d3cfa9a91e3860bb57154c8e02c9a866513d8c WatchSource:0}: Error finding container eb301b9ad8c54454478590a307d3cfa9a91e3860bb57154c8e02c9a866513d8c: Status 404 returned error can't find the container with id eb301b9ad8c54454478590a307d3cfa9a91e3860bb57154c8e02c9a866513d8c
Jul 02 18:31:34.383282 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.383248    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45"]
Jul 02 18:31:34.386441 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.386411    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5wbwr"]
Jul 02 18:31:34.387785 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.387753    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q"]
Jul 02 18:31:34.389882 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.389858    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:31:34.398965 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:34.398934    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05eea021_83b5_4e47_b117_154bb3b30d32.slice/crio-aebdae1c0d30e39a8fb275d111fb11a2a99084e8e6d0fafd946cef5d9f2e0c9b WatchSource:0}: Error finding container aebdae1c0d30e39a8fb275d111fb11a2a99084e8e6d0fafd946cef5d9f2e0c9b: Status 404 returned error can't find the container with id aebdae1c0d30e39a8fb275d111fb11a2a99084e8e6d0fafd946cef5d9f2e0c9b
Jul 02 18:31:34.864956 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.864909    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q" event={"ID":"f0b5acde-5fb9-4fbc-8a51-589ec242aa45","Type":"ContainerStarted","Data":"b047d5a7998ae8b4cdbb599e4463cfd608c89cd45aba9a78d5c22fc0f6b163d5"}
Jul 02 18:31:34.866298 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.866266    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wbwr" event={"ID":"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e","Type":"ContainerStarted","Data":"8db1497e839792923d6345dbf8419fa6a203e9883148aaf2eea24bb160dd952d"}
Jul 02 18:31:34.867827 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.867774    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45" event={"ID":"b21a829d-ec29-4eaf-9f02-5ab7fdb71643","Type":"ContainerStarted","Data":"3b30b7ebd242e286aa3d87a72562f7ad3015e7bb0e9f8b18203f3e184e1ea06b"}
Jul 02 18:31:34.869774 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.869737    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674c7d54dc-s8b5k" event={"ID":"97f2cfcc-f466-4c0f-a731-0adf39a75625","Type":"ContainerStarted","Data":"b57469b5b690cbd03c507977bcf6c736114bd0dd2702f4ab898b4a3151f9d9e7"}
Jul 02 18:31:34.869904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.869795    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-674c7d54dc-s8b5k" event={"ID":"97f2cfcc-f466-4c0f-a731-0adf39a75625","Type":"ContainerStarted","Data":"eb301b9ad8c54454478590a307d3cfa9a91e3860bb57154c8e02c9a866513d8c"}
Jul 02 18:31:34.872129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.871950    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-d5rdt" event={"ID":"07a2791d-f19a-4d3a-9a53-1ec166771a43","Type":"ContainerStarted","Data":"ee164fe733620e1557c48ade235202948921d9910253daf5519a59112d250e32"}
Jul 02 18:31:34.874052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.873885    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" event={"ID":"05eea021-83b5-4e47-b117-154bb3b30d32","Type":"ContainerStarted","Data":"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"}
Jul 02 18:31:34.874052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.873929    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" event={"ID":"05eea021-83b5-4e47-b117-154bb3b30d32","Type":"ContainerStarted","Data":"aebdae1c0d30e39a8fb275d111fb11a2a99084e8e6d0fafd946cef5d9f2e0c9b"}
Jul 02 18:31:34.874185 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.874162    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:34.891007 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.890941    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-674c7d54dc-s8b5k" podStartSLOduration=36.890923024 podStartE2EDuration="36.890923024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:34.889911954 +0000 UTC m=+71.219161578" watchObservedRunningTime="2026-07-02 18:31:34.890923024 +0000 UTC m=+71.220172647"
Jul 02 18:31:34.908452 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.908174    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-d5rdt" podStartSLOduration=66.320931427 podStartE2EDuration="1m9.90815297s" totalImagesPullingTime="3.587221543s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:34.907019106 +0000 UTC m=+71.236268732" watchObservedRunningTime="2026-07-02 18:31:34.90815297 +0000 UTC m=+71.237402594"
Jul 02 18:31:34.928061 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.928001    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" podStartSLOduration=69.927960979 podStartE2EDuration="1m9.927960979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:34.926518706 +0000 UTC m=+71.255768331" watchObservedRunningTime="2026-07-02 18:31:34.927960979 +0000 UTC m=+71.257210603"
Jul 02 18:31:34.984365 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.984316    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:34.987124 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:34.987089    2553 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:35.879009 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:35.878923    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:35.880190 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:35.880166    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-674c7d54dc-s8b5k"
Jul 02 18:31:37.888668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:37.888642    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q" event={"ID":"f0b5acde-5fb9-4fbc-8a51-589ec242aa45","Type":"ContainerStarted","Data":"b0a789e0127400a4af0df8980f40fa8d2989e52866ddf966af004168d4fb7684"}
Jul 02 18:31:37.890434 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:37.890407    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wbwr" event={"ID":"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e","Type":"ContainerStarted","Data":"736c04a222be526d8aa407a150d1137f4001e6d9abc0bf4d400ab0d6b8bf6316"}
Jul 02 18:31:37.892027 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:37.892005    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45" event={"ID":"b21a829d-ec29-4eaf-9f02-5ab7fdb71643","Type":"ContainerStarted","Data":"78f9a60d4e797ce3f7b4b626a7dff6264f0e4616d74f4e8dbc7aad80ad1b244d"}
Jul 02 18:31:38.897492 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.897457    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5wbwr" event={"ID":"93a17f0a-0ba7-45e1-ae42-f87237ee3e7e","Type":"ContainerStarted","Data":"439d7fd2ac264cbc36cbdd1794e81419e2e24619deab47b3400dc9981401578d"}
Jul 02 18:31:38.897900 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.897556    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:38.899286 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.899267    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45" event={"ID":"b21a829d-ec29-4eaf-9f02-5ab7fdb71643","Type":"ContainerStarted","Data":"ae772f77cce585845122367cf238fd95655494d8867feabb9f8255c1f7677f97"}
Jul 02 18:31:38.914995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.914940    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5wbwr" podStartSLOduration=35.575953078 podStartE2EDuration="38.914912973s" totalImagesPullingTime="3.338959895s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:38.913812331 +0000 UTC m=+75.243061954" watchObservedRunningTime="2026-07-02 18:31:38.914912973 +0000 UTC m=+75.244162639"
Jul 02 18:31:38.929723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.929682    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-lhx8q" podStartSLOduration=37.588271027 podStartE2EDuration="40.929668007s" totalImagesPullingTime="3.34139698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:38.928347933 +0000 UTC m=+75.257597557" watchObservedRunningTime="2026-07-02 18:31:38.929668007 +0000 UTC m=+75.258917672"
Jul 02 18:31:38.944013 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:38.943951    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-b7q45" podStartSLOduration=37.633669261 podStartE2EDuration="40.943938955s" totalImagesPullingTime="3.310269694s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:38.943163916 +0000 UTC m=+75.272413539" watchObservedRunningTime="2026-07-02 18:31:38.943938955 +0000 UTC m=+75.273188578"
Jul 02 18:31:41.862052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:41.862012    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:41.864143 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:41.864126    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a844e35-052f-48ba-bb97-675804539124-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zh2zr\" (UID: \"9a844e35-052f-48ba-bb97-675804539124\") " pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:42.080670 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:42.080638    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5tcqj\""
Jul 02 18:31:42.088039 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:42.088012    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zh2zr"
Jul 02 18:31:42.157852 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:42.157823    2553 scope.go:117] "RemoveContainer" containerID="688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72"
Jul 02 18:31:42.158550 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:31:42.158104    2553 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-ctnnf_openshift-console-operator(703b350d-5221-4434-b101-fe395ca946db)\"" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podUID="703b350d-5221-4434-b101-fe395ca946db"
Jul 02 18:31:42.246123 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:42.246062    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zh2zr"]
Jul 02 18:31:42.248485 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:42.248453    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a844e35_052f_48ba_bb97_675804539124.slice/crio-3884057b2323ea0225a8f6301fa59d2891c886586bbba7f99260d37fde88ee2c WatchSource:0}: Error finding container 3884057b2323ea0225a8f6301fa59d2891c886586bbba7f99260d37fde88ee2c: Status 404 returned error can't find the container with id 3884057b2323ea0225a8f6301fa59d2891c886586bbba7f99260d37fde88ee2c
Jul 02 18:31:42.919565 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:42.919517    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2zr" event={"ID":"9a844e35-052f-48ba-bb97-675804539124","Type":"ContainerStarted","Data":"3884057b2323ea0225a8f6301fa59d2891c886586bbba7f99260d37fde88ee2c"}
Jul 02 18:31:46.938045 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:46.938012    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2zr" event={"ID":"9a844e35-052f-48ba-bb97-675804539124","Type":"ContainerStarted","Data":"d49b20f61407e50caa2359740d8858b02ca515c12ada49fe1dd57459e1af98e4"}
Jul 02 18:31:47.944485 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:47.944445    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2zr" event={"ID":"9a844e35-052f-48ba-bb97-675804539124","Type":"ContainerStarted","Data":"a55f9281d2eb7520ff30809cf2d3e3ec13c765c7a2b6760ff8a685b068c6a938"}
Jul 02 18:31:48.905871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:48.905843    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5wbwr"
Jul 02 18:31:49.843345 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.843263    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9rrwt"]
Jul 02 18:31:49.846191 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.846169    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.849171 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.849143    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 18:31:49.849847 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.849676    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 18:31:49.849847 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.849781    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 18:31:49.849847 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.849789    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 18:31:49.850466 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.850447    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vjlgr\""
Jul 02 18:31:49.935202 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935168    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935226    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-textfile\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935273    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-root\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-metrics-client-ca\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935365    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-wtmp\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935404    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skqtw\" (UniqueName: \"kubernetes.io/projected/eb374ce5-64e9-4e57-b999-0617bd1b90a8-kube-api-access-skqtw\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935437    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-tls\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935513    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.935642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.935550    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-sys\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:49.955958 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.955919    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zh2zr" event={"ID":"9a844e35-052f-48ba-bb97-675804539124","Type":"ContainerStarted","Data":"28eb86215362c69d4b15d155c59ac307ba9f3b7322ccb9fe9e9f5d5fe9695f1a"}
Jul 02 18:31:49.975187 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:49.975128    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zh2zr" podStartSLOduration=33.917885771 podStartE2EDuration="40.975106348s" totalImagesPullingTime="7.057220577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:31:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:49.975066114 +0000 UTC m=+86.304315738" watchObservedRunningTime="2026-07-02 18:31:49.975106348 +0000 UTC m=+86.304355973"
Jul 02 18:31:50.036692 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036649    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036703    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-textfile\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036761    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-root\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036788    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-metrics-client-ca\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036829    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-wtmp\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036854    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skqtw\" (UniqueName: \"kubernetes.io/projected/eb374ce5-64e9-4e57-b999-0617bd1b90a8-kube-api-access-skqtw\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.036871    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-root\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037303    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-tls\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037029    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-wtmp\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037392    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037444    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-metrics-client-ca\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037462    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-sys\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.037676 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037509    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb374ce5-64e9-4e57-b999-0617bd1b90a8-sys\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.038172 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037886    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-textfile\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.038172 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.037963    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.039412 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.039391    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.039985 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.039953    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/eb374ce5-64e9-4e57-b999-0617bd1b90a8-node-exporter-tls\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.045782 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.045759    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skqtw\" (UniqueName: \"kubernetes.io/projected/eb374ce5-64e9-4e57-b999-0617bd1b90a8-kube-api-access-skqtw\") pod \"node-exporter-9rrwt\" (UID: \"eb374ce5-64e9-4e57-b999-0617bd1b90a8\") " pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.156792 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.156706    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9rrwt"
Jul 02 18:31:50.168940 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:31:50.168903    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb374ce5_64e9_4e57_b999_0617bd1b90a8.slice/crio-85fe36399f27e5b1b444a95d70b05f315fede2ddc52681cb01bbead7b9390d55 WatchSource:0}: Error finding container 85fe36399f27e5b1b444a95d70b05f315fede2ddc52681cb01bbead7b9390d55: Status 404 returned error can't find the container with id 85fe36399f27e5b1b444a95d70b05f315fede2ddc52681cb01bbead7b9390d55
Jul 02 18:31:50.966873 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:50.966827    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrwt" event={"ID":"eb374ce5-64e9-4e57-b999-0617bd1b90a8","Type":"ContainerStarted","Data":"85fe36399f27e5b1b444a95d70b05f315fede2ddc52681cb01bbead7b9390d55"}
Jul 02 18:31:51.972026 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:51.971967    2553 generic.go:356] "Generic (PLEG): container finished" podID="eb374ce5-64e9-4e57-b999-0617bd1b90a8" containerID="9b38274885310a56d1d9c0bf59129e58417bd13a568eca0a9bb84a9d4237fe72" exitCode=0
Jul 02 18:31:51.972494 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:51.972033    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrwt" event={"ID":"eb374ce5-64e9-4e57-b999-0617bd1b90a8","Type":"ContainerDied","Data":"9b38274885310a56d1d9c0bf59129e58417bd13a568eca0a9bb84a9d4237fe72"}
Jul 02 18:31:52.977983 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:52.977944    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrwt" event={"ID":"eb374ce5-64e9-4e57-b999-0617bd1b90a8","Type":"ContainerStarted","Data":"29e7cf8e037d5bc50c1418a41de83cfa25ed3b7cf29d7608efeee065053cd9a3"}
Jul 02 18:31:52.978504 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:52.978022    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9rrwt" event={"ID":"eb374ce5-64e9-4e57-b999-0617bd1b90a8","Type":"ContainerStarted","Data":"9add6a6aae518410a5546c9398db2e52ccad247e5dd05cce1ee5db536812bca3"}
Jul 02 18:31:53.006958 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:53.006900    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9rrwt" podStartSLOduration=3.058744317 podStartE2EDuration="4.006884488s" totalImagesPullingTime="883.164915ms" totalInitContainerRuntime="64.975256ms" isStatefulPod=true podCreationTimestamp="2026-07-02 18:31:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:53.006511192 +0000 UTC m=+89.335760815" watchObservedRunningTime="2026-07-02 18:31:53.006884488 +0000 UTC m=+89.336134112"
Jul 02 18:31:55.883099 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:55.883070    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:31:57.157057 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:57.157025    2553 scope.go:117] "RemoveContainer" containerID="688f14351f6daf9e8a5bb441f87a729838a696e33925e8635a9afffa71721c72"
Jul 02 18:31:57.665153 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:57.665121    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:31:57.998320 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:57.998239    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:31:57.998465 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:57.998339    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" event={"ID":"703b350d-5221-4434-b101-fe395ca946db","Type":"ContainerStarted","Data":"60ef47aa5e95faf170253297b3f0f7db55a79254ed6d98c9c8d426348c2b483f"}
Jul 02 18:31:57.998624 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:57.998605    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:58.003284 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:58.003260    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf"
Jul 02 18:31:58.024713 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:31:58.024666    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-ctnnf" podStartSLOduration=54.984700467 podStartE2EDuration="1m0.024650383s" totalImagesPullingTime="5.039949916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:30:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:31:58.024216361 +0000 UTC m=+94.353466006" watchObservedRunningTime="2026-07-02 18:31:58.024650383 +0000 UTC m=+94.353900006"
Jul 02 18:32:01.852438 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:01.852408    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kvrxf"
Jul 02 18:32:14.061895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:14.061857    2553 generic.go:356] "Generic (PLEG): container finished" podID="88ac6aaf-06fa-4909-ae0d-14b8849883b8" containerID="454a6e64c915a367dfe7661b367180ba351ec63623146feb5050c29a0f51bb8e" exitCode=0
Jul 02 18:32:14.062348 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:14.061930    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9" event={"ID":"88ac6aaf-06fa-4909-ae0d-14b8849883b8","Type":"ContainerDied","Data":"454a6e64c915a367dfe7661b367180ba351ec63623146feb5050c29a0f51bb8e"}
Jul 02 18:32:14.062348 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:14.062288    2553 scope.go:117] "RemoveContainer" containerID="454a6e64c915a367dfe7661b367180ba351ec63623146feb5050c29a0f51bb8e"
Jul 02 18:32:15.067307 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:15.067272    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-6h2w9" event={"ID":"88ac6aaf-06fa-4909-ae0d-14b8849883b8","Type":"ContainerStarted","Data":"26b59f35de19c0bfa33d1050b4eecb9806d2681a9aa787bfe0c51b4ce6f5bdb3"}
Jul 02 18:32:15.808885 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:15.808855    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-lhx8q_f0b5acde-5fb9-4fbc-8a51-589ec242aa45/cluster-monitoring-operator/0.log"
Jul 02 18:32:17.606946 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:17.606915    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/init-textfile/0.log"
Jul 02 18:32:17.807369 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:17.807341    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/node-exporter/0.log"
Jul 02 18:32:18.006912 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:18.006877    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/kube-rbac-proxy/0.log"
Jul 02 18:32:19.083764 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:19.083733    2553 generic.go:356] "Generic (PLEG): container finished" podID="fe01e806-7bf3-4328-864c-e78edf501e2f" containerID="056cf11aeaeaef51e944e1fe6088d3036a58d8a8123176f87e55b6d5a502cdbe" exitCode=0
Jul 02 18:32:19.084183 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:19.083810    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477" event={"ID":"fe01e806-7bf3-4328-864c-e78edf501e2f","Type":"ContainerDied","Data":"056cf11aeaeaef51e944e1fe6088d3036a58d8a8123176f87e55b6d5a502cdbe"}
Jul 02 18:32:19.084183 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:19.084164    2553 scope.go:117] "RemoveContainer" containerID="056cf11aeaeaef51e944e1fe6088d3036a58d8a8123176f87e55b6d5a502cdbe"
Jul 02 18:32:20.089342 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:20.089305    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-kh477" event={"ID":"fe01e806-7bf3-4328-864c-e78edf501e2f","Type":"ContainerStarted","Data":"c8b6930920048961bfc5fd655a1156242125e1edb0bce898cfb54fe51cddc473"}
Jul 02 18:32:22.606731 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:22.606671    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-b2ss9_1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20/networking-console-plugin/0.log"
Jul 02 18:32:22.684982 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:22.684916    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" podUID="05eea021-83b5-4e47-b117-154bb3b30d32" containerName="registry" containerID="cri-o://d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a" gracePeriod=30
Jul 02 18:32:22.807245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:22.807211    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:32:22.936950 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:22.936921    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:32:23.009864 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.009830    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/3.log"
Jul 02 18:32:23.013044 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013026    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013140 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013064    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013140 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013103    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013215 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013197    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013254 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013239    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013299 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013277    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013348 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013302    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013400    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx2hr\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr\") pod \"05eea021-83b5-4e47-b117-154bb3b30d32\" (UID: \"05eea021-83b5-4e47-b117-154bb3b30d32\") "
Jul 02 18:32:23.013699 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013653    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:32:23.013779 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.013707    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:32:23.015639 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.015595    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:32:23.015738 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.015671    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:32:23.015738 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.015726    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:23.015883 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.015860    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr" (OuterVolumeSpecName: "kube-api-access-kx2hr") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "kube-api-access-kx2hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:23.016216 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.016195    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:32:23.024934 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.024904    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "05eea021-83b5-4e47-b117-154bb3b30d32" (UID: "05eea021-83b5-4e47-b117-154bb3b30d32"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:32:23.102860 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.102820    2553 generic.go:356] "Generic (PLEG): container finished" podID="05eea021-83b5-4e47-b117-154bb3b30d32" containerID="d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a" exitCode=0
Jul 02 18:32:23.103040 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.102871    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6"
Jul 02 18:32:23.103040 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.102872    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" event={"ID":"05eea021-83b5-4e47-b117-154bb3b30d32","Type":"ContainerDied","Data":"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"}
Jul 02 18:32:23.103040 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.102923    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7cddf47b9b-ggzk6" event={"ID":"05eea021-83b5-4e47-b117-154bb3b30d32","Type":"ContainerDied","Data":"aebdae1c0d30e39a8fb275d111fb11a2a99084e8e6d0fafd946cef5d9f2e0c9b"}
Jul 02 18:32:23.103040 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.102949    2553 scope.go:117] "RemoveContainer" containerID="d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"
Jul 02 18:32:23.111482 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.111462    2553 scope.go:117] "RemoveContainer" containerID="d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"
Jul 02 18:32:23.111736 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:32:23.111716    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a\": container with ID starting with d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a not found: ID does not exist" containerID="d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"
Jul 02 18:32:23.111790 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.111746    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a"} err="failed to get container status \"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a\": rpc error: code = NotFound desc = could not find container \"d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a\": container with ID starting with d46df04648480b1cf441615557ae7ae5c7ce0176a2b47a7fa97e9ae20d4ab96a not found: ID does not exist"
Jul 02 18:32:23.114098 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114079    2553 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-trusted-ca\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114103    2553 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/05eea021-83b5-4e47-b117-154bb3b30d32-ca-trust-extracted\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114119    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2hr\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-kube-api-access-kx2hr\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114134    2553 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-bound-sa-token\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114147    2553 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/05eea021-83b5-4e47-b117-154bb3b30d32-registry-certificates\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114161    2553 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-image-registry-private-configuration\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114184 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114175    2553 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/05eea021-83b5-4e47-b117-154bb3b30d32-installation-pull-secrets\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.114366 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.114189    2553 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/05eea021-83b5-4e47-b117-154bb3b30d32-registry-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:32:23.124065 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.124040    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:32:23.127779 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:23.127759    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7cddf47b9b-ggzk6"]
Jul 02 18:32:24.162356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:24.162318    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05eea021-83b5-4e47-b117-154bb3b30d32" path="/var/lib/kubelet/pods/05eea021-83b5-4e47-b117-154bb3b30d32/volumes"
Jul 02 18:32:33.148436 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:33.148398    2553 generic.go:356] "Generic (PLEG): container finished" podID="06c2f72b-be90-44b8-9188-a5f8f32b297c" containerID="7763aecc7d3084c9e34b0dcfd62338b19d6debe202a1cde2fc1decfba30ebb70" exitCode=0
Jul 02 18:32:33.148809 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:33.148471    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j" event={"ID":"06c2f72b-be90-44b8-9188-a5f8f32b297c","Type":"ContainerDied","Data":"7763aecc7d3084c9e34b0dcfd62338b19d6debe202a1cde2fc1decfba30ebb70"}
Jul 02 18:32:33.148853 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:33.148821    2553 scope.go:117] "RemoveContainer" containerID="7763aecc7d3084c9e34b0dcfd62338b19d6debe202a1cde2fc1decfba30ebb70"
Jul 02 18:32:34.162897 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:32:34.162860    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-z669j" event={"ID":"06c2f72b-be90-44b8-9188-a5f8f32b297c","Type":"ContainerStarted","Data":"95f2af0d13ae4d0daac6907ec5bdcf760e9c21d089b92d005e649ade7ae5f75c"}
Jul 02 18:35:06.040185 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.040141    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"]
Jul 02 18:35:06.040701 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.040573    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05eea021-83b5-4e47-b117-154bb3b30d32" containerName="registry"
Jul 02 18:35:06.040701 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.040591    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="05eea021-83b5-4e47-b117-154bb3b30d32" containerName="registry"
Jul 02 18:35:06.040701 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.040688    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="05eea021-83b5-4e47-b117-154bb3b30d32" containerName="registry"
Jul 02 18:35:06.043652 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.043631    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.047690 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.047668    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 18:35:06.047807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.047690    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 18:35:06.047807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.047745    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 18:35:06.047807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.047745    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 02 18:35:06.047996 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.047823    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 02 18:35:06.048066 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.048015    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 02 18:35:06.048122 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.048105    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 02 18:35:06.053714 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.053694    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"]
Jul 02 18:35:06.143253 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143217    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.143443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143272    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.143443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143336    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.143443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143354    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.143443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143393    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xkv9\" (UniqueName: \"kubernetes.io/projected/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-kube-api-access-5xkv9\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.143443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.143421    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244284 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244247    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244464 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244374    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244525 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244507    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244631 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244608    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xkv9\" (UniqueName: \"kubernetes.io/projected/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-kube-api-access-5xkv9\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244693 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244672    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.244749 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.244700    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.245385 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.245362    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.247045 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.247016    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.247129 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.247083    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.247168 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.247140    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-ca\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.247450 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.247428    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.254145 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.254121    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xkv9\" (UniqueName: \"kubernetes.io/projected/bd2538ee-13ff-47aa-a3d2-c6864a886b0b-kube-api-access-5xkv9\") pod \"cluster-proxy-proxy-agent-5dcffd4f-kbvvs\" (UID: \"bd2538ee-13ff-47aa-a3d2-c6864a886b0b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.362695 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.362604    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"
Jul 02 18:35:06.493754 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.493708    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs"]
Jul 02 18:35:06.496393 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:35:06.496367    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd2538ee_13ff_47aa_a3d2_c6864a886b0b.slice/crio-25a9d02edbbc2bf314afa6d29bd3ed32d0b9c1d3b0e4513c64b9c63cd8239fba WatchSource:0}: Error finding container 25a9d02edbbc2bf314afa6d29bd3ed32d0b9c1d3b0e4513c64b9c63cd8239fba: Status 404 returned error can't find the container with id 25a9d02edbbc2bf314afa6d29bd3ed32d0b9c1d3b0e4513c64b9c63cd8239fba
Jul 02 18:35:06.705515 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:06.705484    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs" event={"ID":"bd2538ee-13ff-47aa-a3d2-c6864a886b0b","Type":"ContainerStarted","Data":"25a9d02edbbc2bf314afa6d29bd3ed32d0b9c1d3b0e4513c64b9c63cd8239fba"}
Jul 02 18:35:10.724737 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:10.724703    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs" event={"ID":"bd2538ee-13ff-47aa-a3d2-c6864a886b0b","Type":"ContainerStarted","Data":"b2d53d12383c9d2a0eb7e986f7d0afb8b045fd688ea67f4bcdc0770a261675fa"}
Jul 02 18:35:10.725235 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:10.724752    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs" event={"ID":"bd2538ee-13ff-47aa-a3d2-c6864a886b0b","Type":"ContainerStarted","Data":"c77ac131ae1d6191503d726d003fe74a9d7a9dc66bbc3ab7c1f9583aa2536ae7"}
Jul 02 18:35:10.725235 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:10.724770    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs" event={"ID":"bd2538ee-13ff-47aa-a3d2-c6864a886b0b","Type":"ContainerStarted","Data":"40afc3296e76735557d96b9d4e4d486b5726c0ea2ba06f6ff3513a4570f33bdb"}
Jul 02 18:35:10.743614 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:10.743566    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5dcffd4f-kbvvs" podStartSLOduration=0.727376439 podStartE2EDuration="4.743553213s" totalImagesPullingTime="4.016176774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:35:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:35:10.74324462 +0000 UTC m=+287.072494245" watchObservedRunningTime="2026-07-02 18:35:10.743553213 +0000 UTC m=+287.072802836"
Jul 02 18:35:24.027112 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:24.027084    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:35:24.028614 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:24.028590    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:35:24.035358 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:24.035332    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:35:24.036881 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:24.036863    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:35:24.039127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:35:24.039109    2553 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 18:38:47.504278 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.504244    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-npsnd"]
Jul 02 18:38:47.507289 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.507273    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.509825 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.509803    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 02 18:38:47.510038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.510017    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 18:38:47.511153 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.511131    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x9bd2\""
Jul 02 18:38:47.511260 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.511133    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 18:38:47.513904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.513881    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-npsnd"]
Jul 02 18:38:47.553925 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.553890    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.554142 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.554044    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkbmn\" (UniqueName: \"kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.654860 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.654821    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkbmn\" (UniqueName: \"kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.655022 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.654868    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.655257 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.655240    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.665361 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.665333    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkbmn\" (UniqueName: \"kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn\") pod \"s3-init-npsnd\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") " pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.817552 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.817459    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-npsnd"
Jul 02 18:38:47.943520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.943486    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-npsnd"]
Jul 02 18:38:47.946411 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:38:47.946385    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode70dce61_d9ea_4a5e_9dc6_b48e1e1f4554.slice/crio-f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71 WatchSource:0}: Error finding container f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71: Status 404 returned error can't find the container with id f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71
Jul 02 18:38:47.948349 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:47.948334    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:38:48.519695 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:48.519658    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerStarted","Data":"f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71"}
Jul 02 18:38:52.538327 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:52.538231    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerStarted","Data":"2c094b60bed718a5ffd0ce5997bd5e06b42dc0a601b40f58b35c43177c70b9ea"}
Jul 02 18:38:54.547722 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:54.547689    2553 generic.go:356] "Generic (PLEG): container finished" podID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerID="2c094b60bed718a5ffd0ce5997bd5e06b42dc0a601b40f58b35c43177c70b9ea" exitCode=0
Jul 02 18:38:54.548130 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:38:54.547738    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerDied","Data":"2c094b60bed718a5ffd0ce5997bd5e06b42dc0a601b40f58b35c43177c70b9ea"}
Jul 02 18:39:00.576567 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:39:00.576532    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerStarted","Data":"7a2573ba265537303ed350a791427f8a8f2be18e50f6f0d155e16ded90c78a54"}
Jul 02 18:39:00.592654 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:39:00.592583    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-npsnd" podStartSLOduration=1.755651168 podStartE2EDuration="13.592567261s" totalImagesPullingTime="9.640150438s" totalInitContainerRuntime="2.196765655s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:38:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:39:00.591428755 +0000 UTC m=+516.920678380" watchObservedRunningTime="2026-07-02 18:39:00.592567261 +0000 UTC m=+516.921816883"
Jul 02 18:40:05.821580 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:05.821544    2553 generic.go:356] "Generic (PLEG): container finished" podID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerID="7a2573ba265537303ed350a791427f8a8f2be18e50f6f0d155e16ded90c78a54" exitCode=0
Jul 02 18:40:05.822057 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:05.821593    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerDied","Data":"7a2573ba265537303ed350a791427f8a8f2be18e50f6f0d155e16ded90c78a54"}
Jul 02 18:40:06.949983 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:06.949936    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-npsnd"
Jul 02 18:40:06.961950 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:06.961914    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache\") pod \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") "
Jul 02 18:40:06.962076 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:06.961993    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkbmn\" (UniqueName: \"kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn\") pod \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\" (UID: \"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554\") "
Jul 02 18:40:06.963887 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:06.963865    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn" (OuterVolumeSpecName: "kube-api-access-dkbmn") pod "e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" (UID: "e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554"). InnerVolumeSpecName "kube-api-access-dkbmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:40:06.984356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:06.984330    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache" (OuterVolumeSpecName: "model-cache") pod "e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" (UID: "e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:40:07.062797 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:07.062772    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-model-cache\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:07.062797 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:07.062794    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkbmn\" (UniqueName: \"kubernetes.io/projected/e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554-kube-api-access-dkbmn\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:07.830931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:07.830900    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-npsnd"
Jul 02 18:40:07.830931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:07.830912    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-npsnd" event={"ID":"e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554","Type":"ContainerDied","Data":"f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71"}
Jul 02 18:40:07.831214 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:07.830949    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0e821b54a227f6f48d68b2af4db1afc9e09baea8303538df532c6e19b43da71"
Jul 02 18:40:15.468934 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.468899    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-v68qm"]
Jul 02 18:40:15.469397 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.469178    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerName="download-hf-model"
Jul 02 18:40:15.469397 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.469190    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerName="download-hf-model"
Jul 02 18:40:15.469397 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.469197    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerName="s3-init"
Jul 02 18:40:15.469397 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.469203    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerName="s3-init"
Jul 02 18:40:15.469397 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.469261    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554" containerName="s3-init"
Jul 02 18:40:15.471541 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.471524    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.474200 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.474174    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 02 18:40:15.474200 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.474190    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x9bd2\""
Jul 02 18:40:15.474363 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.474236    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 18:40:15.475686 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.475670    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 18:40:15.478731 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.478712    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-v68qm"]
Jul 02 18:40:15.523132 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.523096    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b9d8\" (UniqueName: \"kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.523289 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.523151    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.624022 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.623964    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b9d8\" (UniqueName: \"kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.624022 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.624030    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.624443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.624422    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.632313 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.632280    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b9d8\" (UniqueName: \"kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8\") pod \"s3-tls-init-custom-v68qm\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") " pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.780914 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.780829    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:15.906124 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:15.906094    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-v68qm"]
Jul 02 18:40:15.909133 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:40:15.909103    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f315b1_e535_4fbe_b5c9_f414fde9f116.slice/crio-c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22 WatchSource:0}: Error finding container c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22: Status 404 returned error can't find the container with id c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22
Jul 02 18:40:16.869853 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:16.869753    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerStarted","Data":"87d016064953be2f0d81e5c6f002239a6580f71f36438fce6a9072e3b58de8d8"}
Jul 02 18:40:16.869853 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:16.869807    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerStarted","Data":"c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22"}
Jul 02 18:40:17.875013 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:17.874954    2553 generic.go:356] "Generic (PLEG): container finished" podID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerID="87d016064953be2f0d81e5c6f002239a6580f71f36438fce6a9072e3b58de8d8" exitCode=0
Jul 02 18:40:17.875387 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:17.875016    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerDied","Data":"87d016064953be2f0d81e5c6f002239a6580f71f36438fce6a9072e3b58de8d8"}
Jul 02 18:40:18.880328 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:18.880293    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerStarted","Data":"da03bb25e331520673c71ac28d2ef0374b5f59b795973a9fd96a50af89ed97c4"}
Jul 02 18:40:18.898538 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:18.898491    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-v68qm" podStartSLOduration=2.152727353 podStartE2EDuration="3.898475313s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.74574796s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:40:18.896867299 +0000 UTC m=+595.226116932" watchObservedRunningTime="2026-07-02 18:40:18.898475313 +0000 UTC m=+595.227724935"
Jul 02 18:40:23.352122 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:40:23.352091    2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f315b1_e535_4fbe_b5c9_f414fde9f116.slice/crio-conmon-da03bb25e331520673c71ac28d2ef0374b5f59b795973a9fd96a50af89ed97c4.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 18:40:23.352482 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:40:23.352103    2553 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f315b1_e535_4fbe_b5c9_f414fde9f116.slice/crio-conmon-da03bb25e331520673c71ac28d2ef0374b5f59b795973a9fd96a50af89ed97c4.scope\": RecentStats: unable to find data in memory cache]"
Jul 02 18:40:23.901523 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:23.901490    2553 generic.go:356] "Generic (PLEG): container finished" podID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerID="da03bb25e331520673c71ac28d2ef0374b5f59b795973a9fd96a50af89ed97c4" exitCode=0
Jul 02 18:40:23.901700 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:23.901564    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerDied","Data":"da03bb25e331520673c71ac28d2ef0374b5f59b795973a9fd96a50af89ed97c4"}
Jul 02 18:40:24.052568 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:24.052542    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:40:24.053728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:24.053705    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:40:24.060261 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:24.060243    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:40:24.061275 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:24.061257    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:40:25.042707 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.042677    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:25.097172 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.097137    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache\") pod \"93f315b1-e535-4fbe-b5c9-f414fde9f116\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") "
Jul 02 18:40:25.097343 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.097209    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9b9d8\" (UniqueName: \"kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8\") pod \"93f315b1-e535-4fbe-b5c9-f414fde9f116\" (UID: \"93f315b1-e535-4fbe-b5c9-f414fde9f116\") "
Jul 02 18:40:25.099438 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.099405    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8" (OuterVolumeSpecName: "kube-api-access-9b9d8") pod "93f315b1-e535-4fbe-b5c9-f414fde9f116" (UID: "93f315b1-e535-4fbe-b5c9-f414fde9f116"). InnerVolumeSpecName "kube-api-access-9b9d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:40:25.119252 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.119215    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache" (OuterVolumeSpecName: "model-cache") pod "93f315b1-e535-4fbe-b5c9-f414fde9f116" (UID: "93f315b1-e535-4fbe-b5c9-f414fde9f116"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:40:25.198465 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.198424    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93f315b1-e535-4fbe-b5c9-f414fde9f116-model-cache\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:25.198465 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.198465    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b9d8\" (UniqueName: \"kubernetes.io/projected/93f315b1-e535-4fbe-b5c9-f414fde9f116-kube-api-access-9b9d8\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:25.910616 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.910581    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-v68qm" event={"ID":"93f315b1-e535-4fbe-b5c9-f414fde9f116","Type":"ContainerDied","Data":"c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22"}
Jul 02 18:40:25.910616 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.910618    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-v68qm"
Jul 02 18:40:25.910862 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:25.910622    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c84ad896aa27f5008747b91b40ed43bde21dfc92b8f02cc86cd518936237fe22"
Jul 02 18:40:29.869324 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869289    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-lnbvt"]
Jul 02 18:40:29.869733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869563    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerName="s3-tls-init-custom"
Jul 02 18:40:29.869733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869575    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerName="s3-tls-init-custom"
Jul 02 18:40:29.869733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869584    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerName="download-hf-model"
Jul 02 18:40:29.869733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869589    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerName="download-hf-model"
Jul 02 18:40:29.869733 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.869649    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="93f315b1-e535-4fbe-b5c9-f414fde9f116" containerName="s3-tls-init-custom"
Jul 02 18:40:29.873429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.873410    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:29.875889 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.875863    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 02 18:40:29.876071 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.875951    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 02 18:40:29.876071 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.876015    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-x9bd2\""
Jul 02 18:40:29.877042 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.877017    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 02 18:40:29.879365 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.879339    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-lnbvt"]
Jul 02 18:40:29.939455 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.939415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:29.939455 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:29.939458    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7rn7\" (UniqueName: \"kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.040595 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.040553    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.040765 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.040604    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7rn7\" (UniqueName: \"kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.041000 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.040943    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.049286 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.049261    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7rn7\" (UniqueName: \"kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7\") pod \"s3-tls-init-serving-lnbvt\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") " pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.183625 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.183593    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:30.314491 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.314446    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-lnbvt"]
Jul 02 18:40:30.317256 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:40:30.317228    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa715faa_69a5_494d_b8fc_2f0b4226abf1.slice/crio-1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e WatchSource:0}: Error finding container 1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e: Status 404 returned error can't find the container with id 1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e
Jul 02 18:40:30.932546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.932511    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerStarted","Data":"635e41978d314bcda0b5b756549180f349172347632cd9ef314831867e6a9be9"}
Jul 02 18:40:30.932940 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:30.932559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerStarted","Data":"1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e"}
Jul 02 18:40:31.938024 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:31.937993    2553 generic.go:356] "Generic (PLEG): container finished" podID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerID="635e41978d314bcda0b5b756549180f349172347632cd9ef314831867e6a9be9" exitCode=0
Jul 02 18:40:31.938336 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:31.938039    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerDied","Data":"635e41978d314bcda0b5b756549180f349172347632cd9ef314831867e6a9be9"}
Jul 02 18:40:32.943743 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:32.943703    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerStarted","Data":"0b3d4c83ddb5626a603a0901b42a0c4f81ca7981ab7789c5ce82cbf45f452be7"}
Jul 02 18:40:32.959537 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:32.959485    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-lnbvt" podStartSLOduration=2.481009675 podStartE2EDuration="3.959468104s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.478458429s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:40:32.95853138 +0000 UTC m=+609.287781002" watchObservedRunningTime="2026-07-02 18:40:32.959468104 +0000 UTC m=+609.288717740"
Jul 02 18:40:39.974440 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:39.974403    2553 generic.go:356] "Generic (PLEG): container finished" podID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerID="0b3d4c83ddb5626a603a0901b42a0c4f81ca7981ab7789c5ce82cbf45f452be7" exitCode=0
Jul 02 18:40:39.974842 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:39.974467    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerDied","Data":"0b3d4c83ddb5626a603a0901b42a0c4f81ca7981ab7789c5ce82cbf45f452be7"}
Jul 02 18:40:41.123104 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.123075    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:41.234927 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.234835    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache\") pod \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") "
Jul 02 18:40:41.235112 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.234942    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7rn7\" (UniqueName: \"kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7\") pod \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\" (UID: \"fa715faa-69a5-494d-b8fc-2f0b4226abf1\") "
Jul 02 18:40:41.236984 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.236944    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7" (OuterVolumeSpecName: "kube-api-access-r7rn7") pod "fa715faa-69a5-494d-b8fc-2f0b4226abf1" (UID: "fa715faa-69a5-494d-b8fc-2f0b4226abf1"). InnerVolumeSpecName "kube-api-access-r7rn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:40:41.257264 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.257223    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa715faa-69a5-494d-b8fc-2f0b4226abf1" (UID: "fa715faa-69a5-494d-b8fc-2f0b4226abf1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:40:41.335507 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.335467    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7rn7\" (UniqueName: \"kubernetes.io/projected/fa715faa-69a5-494d-b8fc-2f0b4226abf1-kube-api-access-r7rn7\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:41.335507 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.335510    2553 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa715faa-69a5-494d-b8fc-2f0b4226abf1-model-cache\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:40:41.984315 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.984285    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-lnbvt"
Jul 02 18:40:41.984513 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.984279    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-lnbvt" event={"ID":"fa715faa-69a5-494d-b8fc-2f0b4226abf1","Type":"ContainerDied","Data":"1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e"}
Jul 02 18:40:41.984513 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:41.984395    2553 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1112cc5575a50544549f167da53c7b2e466218f93f5bf4a4544109900f028f2e"
Jul 02 18:40:54.561850 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.561765    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:40:54.562496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.562107    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerName="download-hf-model"
Jul 02 18:40:54.562496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.562120    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerName="download-hf-model"
Jul 02 18:40:54.562496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.562129    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerName="s3-tls-init-serving"
Jul 02 18:40:54.562496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.562135    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerName="s3-tls-init-serving"
Jul 02 18:40:54.562496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.562186    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa715faa-69a5-494d-b8fc-2f0b4226abf1" containerName="s3-tls-init-serving"
Jul 02 18:40:54.565180 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.565163    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.568056 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.567953    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:40:54.568056 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.567953    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2vv24\""
Jul 02 18:40:54.568056 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.568043    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-kube-rbac-proxy-sar-config\""
Jul 02 18:40:54.568264 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.568120    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:40:54.568264 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.568183    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:40:54.568264 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.568239    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-predictor-serving-cert\""
Jul 02 18:40:54.569022 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.569006    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:40:54.574377 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.574357    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:40:54.643760 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.643733    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.643926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.643794    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.643926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.643813    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.643926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.643850    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m4j6\" (UniqueName: \"kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.643926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.643874    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.744943 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.744910    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.744943 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.744948    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745203 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.744993    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m4j6\" (UniqueName: \"kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745203 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.745124    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745203 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.745176    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745429 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.745411    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745636 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.745608    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.745753 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.745674    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.747727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.747699    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.753224 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.753201    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m4j6\" (UniqueName: \"kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6\") pod \"isvc-sklearn-batcher-predictor-656ff6f494-vfss6\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:54.876814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:54.876726    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:40:55.005518 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:55.005493    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:40:55.007520 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:40:55.007491    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3ed8ca5_6e78_46ff_8805_d9ff748f5a38.slice/crio-97f764ffea1d6788d77f78a95c625452d1c13e83fddbc1b535f4644fcec36ef8 WatchSource:0}: Error finding container 97f764ffea1d6788d77f78a95c625452d1c13e83fddbc1b535f4644fcec36ef8: Status 404 returned error can't find the container with id 97f764ffea1d6788d77f78a95c625452d1c13e83fddbc1b535f4644fcec36ef8
Jul 02 18:40:55.036338 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:55.036312    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerStarted","Data":"97f764ffea1d6788d77f78a95c625452d1c13e83fddbc1b535f4644fcec36ef8"}
Jul 02 18:40:56.041991 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:56.041904    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerStarted","Data":"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"}
Jul 02 18:40:59.054481 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:59.054438    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerID="675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99" exitCode=0
Jul 02 18:40:59.054890 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:40:59.054520    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerDied","Data":"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"}
Jul 02 18:41:13.138155 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:13.138100    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerStarted","Data":"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"}
Jul 02 18:41:16.152709 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:16.152671    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerStarted","Data":"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"}
Jul 02 18:41:19.169408 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:19.169374    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerStarted","Data":"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"}
Jul 02 18:41:19.169885 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:19.169579    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:41:19.189995 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:19.189924    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podStartSLOduration=2.724360074 podStartE2EDuration="25.189910506s" totalImagesPullingTime="18.980806435s" totalInitContainerRuntime="3.484743997s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:40:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:41:19.188367172 +0000 UTC m=+655.517616819" watchObservedRunningTime="2026-07-02 18:41:19.189910506 +0000 UTC m=+655.519160110"
Jul 02 18:41:20.173749 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:20.173716    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:41:20.174187 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:20.173784    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:41:20.175141 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:20.175095    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:20.175829 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:20.175802    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:41:20.179562 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:20.179544    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:41:21.177817 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:21.177775    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:21.178262 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:21.178018    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:41:22.183091 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:22.183042    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:22.183852 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:22.183814    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:41:32.183034 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:32.182992    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:32.183722 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:32.183696    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:41:42.183163 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:42.183108    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:42.183806 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:42.183783    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:41:52.183687 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:52.183645    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:41:52.184126 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:41:52.184098    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:02.183706 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:02.183665    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:42:02.184119 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:02.184098    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:12.184060 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:12.184017    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:42:12.184514 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:12.184484    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:22.184099 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:22.184011    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:42:22.184538 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:22.184516    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:29.158038 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:29.158009    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:42:29.158420 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:29.158153    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:42:39.757203 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.757171    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:42:39.766425 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.766397    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:42:39.766583 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.766520    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.766770 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.766709    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" containerID="cri-o://3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7" gracePeriod=30
Jul 02 18:42:39.767006 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.766775    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" containerID="cri-o://ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df" gracePeriod=30
Jul 02 18:42:39.767122 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.766841    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" containerID="cri-o://ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7" gracePeriod=30
Jul 02 18:42:39.770395 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.770376    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-kube-rbac-proxy-sar-config\""
Jul 02 18:42:39.770531 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.770413    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-batcher-custom-predictor-serving-cert\""
Jul 02 18:42:39.778455 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.778432    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:42:39.886176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.886152    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.886272 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.886192    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.886272 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.886242    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.886272 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.886260    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfsg\" (UniqueName: \"kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.886389 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.886307    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.986984 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.986952    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987005    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987113    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfsg\" (UniqueName: \"kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987283 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987173    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987283 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987255    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987496 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987476    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987726    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.987807 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.987778    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.989592 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.989571    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:39.995530 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:39.995508    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfsg\" (UniqueName: \"kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg\") pod \"isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") " pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:40.086653 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.086582    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:40.174555 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.174512    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:42:40.213578 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.213556    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:42:40.215516 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:42:40.215492    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf90f810b_1431_4c99_ba3b_8ef625368aa8.slice/crio-fda56ba9fd430ec4b8605cf104481f32f349caa2b21174363977b65b2e44de94 WatchSource:0}: Error finding container fda56ba9fd430ec4b8605cf104481f32f349caa2b21174363977b65b2e44de94: Status 404 returned error can't find the container with id fda56ba9fd430ec4b8605cf104481f32f349caa2b21174363977b65b2e44de94
Jul 02 18:42:40.496909 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.496871    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerID="ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7" exitCode=2
Jul 02 18:42:40.497075 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.496938    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerDied","Data":"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"}
Jul 02 18:42:40.498451 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.498422    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerStarted","Data":"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"}
Jul 02 18:42:40.498577 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:40.498492    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerStarted","Data":"fda56ba9fd430ec4b8605cf104481f32f349caa2b21174363977b65b2e44de94"}
Jul 02 18:42:44.518991 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:44.518942    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerID="3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7" exitCode=0
Jul 02 18:42:44.519421 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:44.519018    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerDied","Data":"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"}
Jul 02 18:42:44.520356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:44.520336    2553 generic.go:356] "Generic (PLEG): container finished" podID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerID="3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c" exitCode=0
Jul 02 18:42:44.520446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:44.520376    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerDied","Data":"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"}
Jul 02 18:42:45.174668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.174629    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:42:45.527262 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.527185    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerStarted","Data":"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"}
Jul 02 18:42:45.527262 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.527247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerStarted","Data":"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"}
Jul 02 18:42:45.527627 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.527272    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerStarted","Data":"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"}
Jul 02 18:42:45.527627 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.527553    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:45.527719 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.527699    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:45.529064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.529039    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:42:45.548228 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:45.548190    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podStartSLOduration=2.749858566 podStartE2EDuration="6.548177221s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.798318655s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:42:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:42:45.546769502 +0000 UTC m=+741.876019142" watchObservedRunningTime="2026-07-02 18:42:45.548177221 +0000 UTC m=+741.877426844"
Jul 02 18:42:46.532580 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:46.532546    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:46.532959 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:46.532688    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:42:46.533744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:46.533717    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:47.536703 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:47.536645    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:42:47.537174 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:47.537149    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:49.157469 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:49.157426    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:42:49.157920 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:49.157732    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:50.174194 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:50.174146    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:42:50.174697 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:50.174256    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:42:52.541464 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:52.541431    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:42:52.542210 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:52.542181    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:42:52.542436 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:52.542406    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:42:55.174334 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:55.174291    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:42:59.158405 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:59.158356    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:42:59.158846 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:42:59.158720    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:00.174462 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:00.174420    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:43:02.542127 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:02.542088    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:02.542553 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:02.542532    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:05.174379 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:05.174336    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.26:8643/healthz\": dial tcp 10.132.0.26:8643: connect: connection refused"
Jul 02 18:43:09.157937 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:09.157882    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.26:8080: connect: connection refused"
Jul 02 18:43:09.158423 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:09.158081    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:43:09.158423 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:09.158253    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:09.158423 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:09.158385    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:43:09.914403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:09.914377    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:43:10.041405 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041303    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls\") pod \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") "
Jul 02 18:43:10.041405 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041367    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config\") pod \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") "
Jul 02 18:43:10.041405 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041395    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location\") pod \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") "
Jul 02 18:43:10.041673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041414    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m4j6\" (UniqueName: \"kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6\") pod \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") "
Jul 02 18:43:10.041673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041486    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert\") pod \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\" (UID: \"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38\") "
Jul 02 18:43:10.041865 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041769    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" (UID: "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:43:10.041941 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041773    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" (UID: "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:43:10.042012 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.041940    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" (UID: "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:43:10.043637 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.043603    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" (UID: "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:43:10.043738 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.043632    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6" (OuterVolumeSpecName: "kube-api-access-4m4j6") pod "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" (UID: "b3ed8ca5-6e78-46ff-8805-d9ff748f5a38"). InnerVolumeSpecName "kube-api-access-4m4j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:43:10.142278 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.142242    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:43:10.142278 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.142271    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:43:10.142278 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.142281    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:43:10.142515 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.142292    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:43:10.142515 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.142301    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m4j6\" (UniqueName: \"kubernetes.io/projected/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38-kube-api-access-4m4j6\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:43:10.643501 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.643463    2553 generic.go:356] "Generic (PLEG): container finished" podID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerID="ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df" exitCode=0
Jul 02 18:43:10.644005 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.643505    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerDied","Data":"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"}
Jul 02 18:43:10.644005 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.643520    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"
Jul 02 18:43:10.644005 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.643587    2553 scope.go:117] "RemoveContainer" containerID="ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"
Jul 02 18:43:10.644005 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.643576    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6" event={"ID":"b3ed8ca5-6e78-46ff-8805-d9ff748f5a38","Type":"ContainerDied","Data":"97f764ffea1d6788d77f78a95c625452d1c13e83fddbc1b535f4644fcec36ef8"}
Jul 02 18:43:10.651817 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.651797    2553 scope.go:117] "RemoveContainer" containerID="ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"
Jul 02 18:43:10.659395 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.659376    2553 scope.go:117] "RemoveContainer" containerID="3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"
Jul 02 18:43:10.664279 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.664255    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:43:10.667335 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.667315    2553 scope.go:117] "RemoveContainer" containerID="675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"
Jul 02 18:43:10.667665 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.667646    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-predictor-656ff6f494-vfss6"]
Jul 02 18:43:10.675195 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.675176    2553 scope.go:117] "RemoveContainer" containerID="ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"
Jul 02 18:43:10.675473 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:43:10.675454    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df\": container with ID starting with ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df not found: ID does not exist" containerID="ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"
Jul 02 18:43:10.675537 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.675480    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df"} err="failed to get container status \"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df\": rpc error: code = NotFound desc = could not find container \"ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df\": container with ID starting with ef0ac22e3cf053471a7f0da192f4644ad365b3048cef621b1df43baaf6a150df not found: ID does not exist"
Jul 02 18:43:10.675537 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.675499    2553 scope.go:117] "RemoveContainer" containerID="ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"
Jul 02 18:43:10.675776 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:43:10.675756    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7\": container with ID starting with ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7 not found: ID does not exist" containerID="ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"
Jul 02 18:43:10.675822 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.675783    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7"} err="failed to get container status \"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7\": rpc error: code = NotFound desc = could not find container \"ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7\": container with ID starting with ad7ff0fd5f51d8bf8a39dab86cb6ce8bdb70a555f10545ea0ecd4ccda7656ef7 not found: ID does not exist"
Jul 02 18:43:10.675822 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.675801    2553 scope.go:117] "RemoveContainer" containerID="3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"
Jul 02 18:43:10.676059 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:43:10.676037    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7\": container with ID starting with 3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7 not found: ID does not exist" containerID="3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"
Jul 02 18:43:10.676108 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.676068    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7"} err="failed to get container status \"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7\": rpc error: code = NotFound desc = could not find container \"3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7\": container with ID starting with 3281564adad98384330964be0d6c6e91fc82be7a2779f391ef1915ace77f63e7 not found: ID does not exist"
Jul 02 18:43:10.676108 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.676088    2553 scope.go:117] "RemoveContainer" containerID="675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"
Jul 02 18:43:10.676324 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:43:10.676306    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99\": container with ID starting with 675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99 not found: ID does not exist" containerID="675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"
Jul 02 18:43:10.676374 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:10.676329    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99"} err="failed to get container status \"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99\": rpc error: code = NotFound desc = could not find container \"675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99\": container with ID starting with 675998ac8434688aa95580c8f59ce243b9821df83ec51ad39e69caf72384be99 not found: ID does not exist"
Jul 02 18:43:12.161428 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:12.161397    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" path="/var/lib/kubelet/pods/b3ed8ca5-6e78-46ff-8805-d9ff748f5a38/volumes"
Jul 02 18:43:12.542852 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:12.542753    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:12.543279 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:12.543252    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:22.542033 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:22.541992    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:22.542526 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:22.542375    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:32.542546 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:32.542491    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:32.542990 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:32.542908    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:42.542612 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:42.542572    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:42.543086 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:42.543067    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:43:52.542671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:52.542576    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:43:52.543128 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:43:52.543101    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:02.543185 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:02.543151    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:02.543652 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:02.543323    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:14.845187 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845155    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845480    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845492    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845500    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845506    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845518    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845525    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845540    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="storage-initializer"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845546    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="storage-initializer"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845596    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kserve-container"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845610    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="agent"
Jul 02 18:44:14.845671 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.845619    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3ed8ca5-6e78-46ff-8805-d9ff748f5a38" containerName="kube-rbac-proxy"
Jul 02 18:44:14.847821 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.847799    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:14.850426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.850401    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 02 18:44:14.850426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.850406    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 02 18:44:14.858871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.858838    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:44:14.859226 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.859180    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" containerID="cri-o://cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c" gracePeriod=30
Jul 02 18:44:14.859390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.859310    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" containerID="cri-o://0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2" gracePeriod=30
Jul 02 18:44:14.859508 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.859249    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" containerID="cri-o://c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548" gracePeriod=30
Jul 02 18:44:14.861052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.860968    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:44:14.954403 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.954372    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:14.954554 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.954410    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwvxh\" (UniqueName: \"kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:14.954554 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:14.954450    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.055369 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.055343    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwvxh\" (UniqueName: \"kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.055500 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.055382    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.055500 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.055461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.056083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.056062    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.057790 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.057772    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.063550 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.063521    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwvxh\" (UniqueName: \"kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh\") pod \"message-dumper-predictor-6575bcfdfd-nrsj8\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.160426 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.160396    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:15.289157 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.288962    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:44:15.291686 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:44:15.291649    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a03d050_1bae_4721_a607_4ed603b09f4f.slice/crio-0b07bf19a78fd5e9cf19daf11b8c16804a5d87a9e725e4bfca460268395566a4 WatchSource:0}: Error finding container 0b07bf19a78fd5e9cf19daf11b8c16804a5d87a9e725e4bfca460268395566a4: Status 404 returned error can't find the container with id 0b07bf19a78fd5e9cf19daf11b8c16804a5d87a9e725e4bfca460268395566a4
Jul 02 18:44:15.293498 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.293482    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:44:15.901736 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.901705    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerStarted","Data":"0b07bf19a78fd5e9cf19daf11b8c16804a5d87a9e725e4bfca460268395566a4"}
Jul 02 18:44:15.904229 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.904201    2553 generic.go:356] "Generic (PLEG): container finished" podID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerID="c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548" exitCode=2
Jul 02 18:44:15.904341 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:15.904278    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerDied","Data":"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"}
Jul 02 18:44:16.910100 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:16.909998    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerStarted","Data":"4a19daf59314985b375ce679bf78fce00463f34dd8949ec672ae7a7fc6684c8d"}
Jul 02 18:44:16.910100 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:16.910064    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerStarted","Data":"c1093106073839fa842eed77ac7d4fcc653625c6f3ae8bd8e808bb965e17369c"}
Jul 02 18:44:16.910583 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:16.910102    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:16.928305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:16.928261    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" podStartSLOduration=1.572068077 podStartE2EDuration="2.928247055s" totalImagesPullingTime="1.356178978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:44:16.926071847 +0000 UTC m=+833.255321496" watchObservedRunningTime="2026-07-02 18:44:16.928247055 +0000 UTC m=+833.257496678"
Jul 02 18:44:17.537321 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:17.537285    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:17.914088 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:17.914066    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:17.915849 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:17.915823    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:19.924396 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:19.924362    2553 generic.go:356] "Generic (PLEG): container finished" podID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerID="cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c" exitCode=0
Jul 02 18:44:19.924782 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:19.924440    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerDied","Data":"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"}
Jul 02 18:44:22.536832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:22.536788    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:22.542309 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:22.542284    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:44:22.542609 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:22.542587    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:24.928904 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:24.928870    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:44:27.537207 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:27.537168    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:27.537581 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:27.537298    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:32.537047 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:32.537002    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:32.542404 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:32.542380    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:44:32.542744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:32.542720    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:34.916515 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:34.916483    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:44:34.920183 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:34.920164    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:34.922744 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:34.922719    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-predictor-serving-cert\""
Jul 02 18:44:34.922865 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:34.922780    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-kube-rbac-proxy-sar-config\""
Jul 02 18:44:34.930268 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:34.930246    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:44:35.013236 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.013200    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.013390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.013242    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.013390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.013303    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.013390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.013324    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-672kq\" (UniqueName: \"kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.013390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.013374    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114113 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114081    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114127    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114153    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114183    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114244 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114201    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-672kq\" (UniqueName: \"kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.114726 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.114699    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.115054 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.115030    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.115153 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.115030    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.116560 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.116529    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.122540 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.122516    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-672kq\" (UniqueName: \"kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq\") pod \"isvc-logger-predictor-6697b4c4cc-vn7wz\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") " pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.231917 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.231830    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:35.368030 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.368006    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:44:35.370087 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:44:35.370057    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c1a5de8_ba6f_4af0_bdae_1ce0ed67354c.slice/crio-5c38b39edd4e291e73acbfb26fb80848c9e40b77b493be8d5b12efc940e9138a WatchSource:0}: Error finding container 5c38b39edd4e291e73acbfb26fb80848c9e40b77b493be8d5b12efc940e9138a: Status 404 returned error can't find the container with id 5c38b39edd4e291e73acbfb26fb80848c9e40b77b493be8d5b12efc940e9138a
Jul 02 18:44:35.991743 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.991709    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerStarted","Data":"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"}
Jul 02 18:44:35.992160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:35.991771    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerStarted","Data":"5c38b39edd4e291e73acbfb26fb80848c9e40b77b493be8d5b12efc940e9138a"}
Jul 02 18:44:37.536988 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:37.536929    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:39.006098 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:39.006070    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerID="45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a" exitCode=0
Jul 02 18:44:39.006472 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:39.006150    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerDied","Data":"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"}
Jul 02 18:44:40.013551 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.013518    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerStarted","Data":"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"}
Jul 02 18:44:40.014009 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.013583    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerStarted","Data":"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"}
Jul 02 18:44:40.014009 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.013607    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerStarted","Data":"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"}
Jul 02 18:44:40.014009 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.013895    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:40.014009 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.013963    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:40.014166 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.014022    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:40.015467 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.015442    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:44:40.016121 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.016103    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:40.037242 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:40.037198    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podStartSLOduration=2.579967582 podStartE2EDuration="6.03718564s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.457218058s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:44:40.035388612 +0000 UTC m=+856.364638265" watchObservedRunningTime="2026-07-02 18:44:40.03718564 +0000 UTC m=+856.366435263"
Jul 02 18:44:41.019459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:41.019414    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:44:41.019889 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:41.019867    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:42.536897 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:42.536853    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.27:8643/healthz\": dial tcp 10.132.0.27:8643: connect: connection refused"
Jul 02 18:44:42.542211 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:42.542170    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.27:5000: connect: connection refused"
Jul 02 18:44:42.542353 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:42.542332    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:42.542550 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:42.542531    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:42.542628 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:42.542616    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:45.031966 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.031943    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:45.040410 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.040386    2553 generic.go:356] "Generic (PLEG): container finished" podID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerID="0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2" exitCode=0
Jul 02 18:44:45.040514 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.040437    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"
Jul 02 18:44:45.040514 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.040471    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerDied","Data":"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"}
Jul 02 18:44:45.040594 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.040542    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c" event={"ID":"f90f810b-1431-4c99-ba3b-8ef625368aa8","Type":"ContainerDied","Data":"fda56ba9fd430ec4b8605cf104481f32f349caa2b21174363977b65b2e44de94"}
Jul 02 18:44:45.040635 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.040590    2553 scope.go:117] "RemoveContainer" containerID="0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"
Jul 02 18:44:45.048018 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.048003    2553 scope.go:117] "RemoveContainer" containerID="c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"
Jul 02 18:44:45.056678 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.056656    2553 scope.go:117] "RemoveContainer" containerID="cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"
Jul 02 18:44:45.064071 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.064051    2553 scope.go:117] "RemoveContainer" containerID="3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"
Jul 02 18:44:45.073240 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.073139    2553 scope.go:117] "RemoveContainer" containerID="0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"
Jul 02 18:44:45.073429 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:44:45.073409    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2\": container with ID starting with 0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2 not found: ID does not exist" containerID="0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"
Jul 02 18:44:45.073531 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.073436    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2"} err="failed to get container status \"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2\": rpc error: code = NotFound desc = could not find container \"0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2\": container with ID starting with 0f4ef32fb4bd3f6edbd7b3a23648b272890e0d956f227b8b84fc468a2562ced2 not found: ID does not exist"
Jul 02 18:44:45.073531 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.073453    2553 scope.go:117] "RemoveContainer" containerID="c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"
Jul 02 18:44:45.073718 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:44:45.073699    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548\": container with ID starting with c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548 not found: ID does not exist" containerID="c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"
Jul 02 18:44:45.073781 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.073722    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548"} err="failed to get container status \"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548\": rpc error: code = NotFound desc = could not find container \"c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548\": container with ID starting with c7672ccd444aeed7b531cd0d66e0a0b9ff05c6c4437af61d3d44e3e5e971a548 not found: ID does not exist"
Jul 02 18:44:45.073781 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.073735    2553 scope.go:117] "RemoveContainer" containerID="cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"
Jul 02 18:44:45.074002 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:44:45.073965    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c\": container with ID starting with cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c not found: ID does not exist" containerID="cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"
Jul 02 18:44:45.074077 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.074005    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c"} err="failed to get container status \"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c\": rpc error: code = NotFound desc = could not find container \"cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c\": container with ID starting with cace90c2401030da7161f743e850b7439e990b8179b801c77498a294d382c57c not found: ID does not exist"
Jul 02 18:44:45.074077 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.074018    2553 scope.go:117] "RemoveContainer" containerID="3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"
Jul 02 18:44:45.074220 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:44:45.074205    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c\": container with ID starting with 3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c not found: ID does not exist" containerID="3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"
Jul 02 18:44:45.074280 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.074221    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c"} err="failed to get container status \"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c\": rpc error: code = NotFound desc = could not find container \"3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c\": container with ID starting with 3c53af0ed5f50591fffbf491f594982710b8b85736ea0ce3c42224355533f84c not found: ID does not exist"
Jul 02 18:44:45.097081 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097060    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgfsg\" (UniqueName: \"kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg\") pod \"f90f810b-1431-4c99-ba3b-8ef625368aa8\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") "
Jul 02 18:44:45.097167 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097087    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location\") pod \"f90f810b-1431-4c99-ba3b-8ef625368aa8\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") "
Jul 02 18:44:45.097167 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097116    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls\") pod \"f90f810b-1431-4c99-ba3b-8ef625368aa8\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") "
Jul 02 18:44:45.097275 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097252    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config\") pod \"f90f810b-1431-4c99-ba3b-8ef625368aa8\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") "
Jul 02 18:44:45.097357 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097333    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f90f810b-1431-4c99-ba3b-8ef625368aa8" (UID: "f90f810b-1431-4c99-ba3b-8ef625368aa8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:44:45.097357 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097345    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert\") pod \"f90f810b-1431-4c99-ba3b-8ef625368aa8\" (UID: \"f90f810b-1431-4c99-ba3b-8ef625368aa8\") "
Jul 02 18:44:45.097613 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097586    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f90f810b-1431-4c99-ba3b-8ef625368aa8" (UID: "f90f810b-1431-4c99-ba3b-8ef625368aa8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:44:45.097613 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097595    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f90f810b-1431-4c99-ba3b-8ef625368aa8" (UID: "f90f810b-1431-4c99-ba3b-8ef625368aa8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:44:45.097749 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.097593    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f90f810b-1431-4c99-ba3b-8ef625368aa8-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:44:45.099032 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.099012    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg" (OuterVolumeSpecName: "kube-api-access-jgfsg") pod "f90f810b-1431-4c99-ba3b-8ef625368aa8" (UID: "f90f810b-1431-4c99-ba3b-8ef625368aa8"). InnerVolumeSpecName "kube-api-access-jgfsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:44:45.099119 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.099013    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f90f810b-1431-4c99-ba3b-8ef625368aa8" (UID: "f90f810b-1431-4c99-ba3b-8ef625368aa8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:44:45.198480 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.198448    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:44:45.198480 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.198479    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f90f810b-1431-4c99-ba3b-8ef625368aa8-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:44:45.198661 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.198494    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgfsg\" (UniqueName: \"kubernetes.io/projected/f90f810b-1431-4c99-ba3b-8ef625368aa8-kube-api-access-jgfsg\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:44:45.198661 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.198507    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f90f810b-1431-4c99-ba3b-8ef625368aa8-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:44:45.362479 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.362446    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:44:45.366731 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:45.366701    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-batcher-custom-predictor-7779c66d45-phf9c"]
Jul 02 18:44:46.024306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:46.024275    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:44:46.024868 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:46.024834    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:44:46.025139 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:46.025115    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:44:46.161386 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:46.161353    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" path="/var/lib/kubelet/pods/f90f810b-1431-4c99-ba3b-8ef625368aa8/volumes"
Jul 02 18:44:56.025141 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:56.025102    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:44:56.025590 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:44:56.025561    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:06.024897 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:06.024846    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:45:06.025364 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:06.025296    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:16.025047 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:16.024917    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:45:16.025479 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:16.025403    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:24.079202 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:24.079168    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:45:24.081996 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:24.081957    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:45:24.085966 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:24.085948    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:45:24.088870 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:24.088852    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:45:26.024826 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:26.024782    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:45:26.025226 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:26.025153    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:36.025116 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:36.025060    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:45:36.025570 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:36.025490    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:46.025234 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:46.025188    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:45:46.025720 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:46.025595    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:45:56.026176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:56.026142    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:45:56.026567 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:45:56.026293    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:09.945457 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:09.945424    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-nrsj8_0a03d050-1bae-4721-a607-4ed603b09f4f/kserve-container/0.log"
Jul 02 18:46:10.166482 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.166447    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:46:10.166723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.166696    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" containerID="cri-o://cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a" gracePeriod=30
Jul 02 18:46:10.166818 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.166743    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" containerID="cri-o://510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479" gracePeriod=30
Jul 02 18:46:10.166818 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.166793    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" containerID="cri-o://6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0" gracePeriod=30
Jul 02 18:46:10.220144 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.220070    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:46:10.220323 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.220285    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kserve-container" containerID="cri-o://c1093106073839fa842eed77ac7d4fcc653625c6f3ae8bd8e808bb965e17369c" gracePeriod=30
Jul 02 18:46:10.220415 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.220331    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kube-rbac-proxy" containerID="cri-o://4a19daf59314985b375ce679bf78fce00463f34dd8949ec672ae7a7fc6684c8d" gracePeriod=30
Jul 02 18:46:10.423762 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.423725    2553 generic.go:356] "Generic (PLEG): container finished" podID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerID="4a19daf59314985b375ce679bf78fce00463f34dd8949ec672ae7a7fc6684c8d" exitCode=2
Jul 02 18:46:10.423762 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.423756    2553 generic.go:356] "Generic (PLEG): container finished" podID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerID="c1093106073839fa842eed77ac7d4fcc653625c6f3ae8bd8e808bb965e17369c" exitCode=2
Jul 02 18:46:10.423967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.423793    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerDied","Data":"4a19daf59314985b375ce679bf78fce00463f34dd8949ec672ae7a7fc6684c8d"}
Jul 02 18:46:10.423967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.423851    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerDied","Data":"c1093106073839fa842eed77ac7d4fcc653625c6f3ae8bd8e808bb965e17369c"}
Jul 02 18:46:10.426322 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.426297    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerID="6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0" exitCode=2
Jul 02 18:46:10.426437 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.426360    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerDied","Data":"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"}
Jul 02 18:46:10.468661 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.468632    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:46:10.603564 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.603459    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config\") pod \"0a03d050-1bae-4721-a607-4ed603b09f4f\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") "
Jul 02 18:46:10.603564 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.603508    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lwvxh\" (UniqueName: \"kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh\") pod \"0a03d050-1bae-4721-a607-4ed603b09f4f\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") "
Jul 02 18:46:10.603813 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.603609    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls\") pod \"0a03d050-1bae-4721-a607-4ed603b09f4f\" (UID: \"0a03d050-1bae-4721-a607-4ed603b09f4f\") "
Jul 02 18:46:10.603907 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.603882    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0a03d050-1bae-4721-a607-4ed603b09f4f" (UID: "0a03d050-1bae-4721-a607-4ed603b09f4f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:46:10.605673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.605642    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh" (OuterVolumeSpecName: "kube-api-access-lwvxh") pod "0a03d050-1bae-4721-a607-4ed603b09f4f" (UID: "0a03d050-1bae-4721-a607-4ed603b09f4f"). InnerVolumeSpecName "kube-api-access-lwvxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:46:10.605766 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.605729    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0a03d050-1bae-4721-a607-4ed603b09f4f" (UID: "0a03d050-1bae-4721-a607-4ed603b09f4f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:46:10.704223 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.704185    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0a03d050-1bae-4721-a607-4ed603b09f4f-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:10.704223 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.704217    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:10.704223 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:10.704229    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwvxh\" (UniqueName: \"kubernetes.io/projected/0a03d050-1bae-4721-a607-4ed603b09f4f-kube-api-access-lwvxh\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:11.020360 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.020320    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:11.432211 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.432184    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"
Jul 02 18:46:11.432380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.432177    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8" event={"ID":"0a03d050-1bae-4721-a607-4ed603b09f4f","Type":"ContainerDied","Data":"0b07bf19a78fd5e9cf19daf11b8c16804a5d87a9e725e4bfca460268395566a4"}
Jul 02 18:46:11.432380 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.432327    2553 scope.go:117] "RemoveContainer" containerID="4a19daf59314985b375ce679bf78fce00463f34dd8949ec672ae7a7fc6684c8d"
Jul 02 18:46:11.440947 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.440930    2553 scope.go:117] "RemoveContainer" containerID="c1093106073839fa842eed77ac7d4fcc653625c6f3ae8bd8e808bb965e17369c"
Jul 02 18:46:11.453762 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.453737    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:46:11.456864 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:11.456844    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-nrsj8"]
Jul 02 18:46:12.161786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:12.161754    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" path="/var/lib/kubelet/pods/0a03d050-1bae-4721-a607-4ed603b09f4f/volumes"
Jul 02 18:46:15.454388 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:15.454355    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerID="cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a" exitCode=0
Jul 02 18:46:15.454789 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:15.454403    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerDied","Data":"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"}
Jul 02 18:46:16.020217 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:16.020173    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:16.025645 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:16.025608    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:46:16.026082 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:16.026058    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:46:20.204723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.204692    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.204994    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205006    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205016    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205021    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205030    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205036    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205045    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="storage-initializer"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205050    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="storage-initializer"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205058    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205065    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205074    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kube-rbac-proxy"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205079    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kube-rbac-proxy"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205128    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205136    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kube-rbac-proxy"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205141    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="kserve-container"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205147    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="f90f810b-1431-4c99-ba3b-8ef625368aa8" containerName="agent"
Jul 02 18:46:20.205221 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.205153    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a03d050-1bae-4721-a607-4ed603b09f4f" containerName="kube-rbac-proxy"
Jul 02 18:46:20.208283 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.208266    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.210816 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.210795    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-kube-rbac-proxy-sar-config\""
Jul 02 18:46:20.210963 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.210863    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v1-predictor-serving-cert\""
Jul 02 18:46:20.220574 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.220551    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:46:20.285965 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.285924    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.286189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.285998    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9krpk\" (UniqueName: \"kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.286189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.286027    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.286189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.286048    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.286189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.286118    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.386498 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.386456    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.386668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.386524    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9krpk\" (UniqueName: \"kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.386668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.386583    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.386668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.386617    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.386668 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.386650    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.387043 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.387016    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.387270 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.387249    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.387270 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.387258    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.389061 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.389043    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.395104 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.395086    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9krpk\" (UniqueName: \"kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk\") pod \"isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") " pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.518022 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.517898    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:46:20.645520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:20.645494    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:46:20.648125 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:46:20.648100    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74fc5a63_cf19_4b35_98e6_cba16ebd1db6.slice/crio-b7ef91b7de6eac877bbce201c67b2bb0f1cd90d5eaf67f052e7bad0a70a8ae96 WatchSource:0}: Error finding container b7ef91b7de6eac877bbce201c67b2bb0f1cd90d5eaf67f052e7bad0a70a8ae96: Status 404 returned error can't find the container with id b7ef91b7de6eac877bbce201c67b2bb0f1cd90d5eaf67f052e7bad0a70a8ae96
Jul 02 18:46:21.019836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:21.019792    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:21.020013 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:21.019946    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:21.480831 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:21.480789    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerStarted","Data":"79dd951735875c919d0de276fd07099ee7afc39afce6d3209912855836434974"}
Jul 02 18:46:21.481245 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:21.480895    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerStarted","Data":"b7ef91b7de6eac877bbce201c67b2bb0f1cd90d5eaf67f052e7bad0a70a8ae96"}
Jul 02 18:46:26.020490 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:26.020443    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:26.025026 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:26.024995    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:46:26.025390 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:26.025365    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:46:31.019931 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:31.019888    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:36.020400 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:36.020361    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.132.0.29:8643/healthz\": dial tcp 10.132.0.29:8643: connect: connection refused"
Jul 02 18:46:36.025814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:36.025784    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.29:8080: connect: connection refused"
Jul 02 18:46:36.025922 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:36.025906    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:36.026167 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:36.026145    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 18:46:36.026268 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:36.026255    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:40.309373 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.309344    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:40.453803 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.453708    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config\") pod \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") "
Jul 02 18:46:40.453803 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.453769    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-672kq\" (UniqueName: \"kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq\") pod \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") "
Jul 02 18:46:40.454092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.453809    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls\") pod \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") "
Jul 02 18:46:40.454092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.453859    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location\") pod \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") "
Jul 02 18:46:40.454092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.453892    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert\") pod \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\" (UID: \"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c\") "
Jul 02 18:46:40.454294 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.454239    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" (UID: "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:46:40.454341 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.454311    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" (UID: "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:46:40.454424 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.454406    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" (UID: "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:46:40.455865 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.455843    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq" (OuterVolumeSpecName: "kube-api-access-672kq") pod "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" (UID: "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c"). InnerVolumeSpecName "kube-api-access-672kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:46:40.455964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.455844    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" (UID: "1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:46:40.555302 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.555264    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:40.555802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.555486    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-672kq\" (UniqueName: \"kubernetes.io/projected/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kube-api-access-672kq\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:40.555802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.555507    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:40.555802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.555533    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:40.555802 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.555546    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:46:40.565218 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.565180    2553 generic.go:356] "Generic (PLEG): container finished" podID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerID="510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479" exitCode=0
Jul 02 18:46:40.565349 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.565308    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerDied","Data":"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"}
Jul 02 18:46:40.565415 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.565326    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"
Jul 02 18:46:40.565464 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.565429    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz" event={"ID":"1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c","Type":"ContainerDied","Data":"5c38b39edd4e291e73acbfb26fb80848c9e40b77b493be8d5b12efc940e9138a"}
Jul 02 18:46:40.565500 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.565481    2553 scope.go:117] "RemoveContainer" containerID="510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"
Jul 02 18:46:40.574211 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.574190    2553 scope.go:117] "RemoveContainer" containerID="6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"
Jul 02 18:46:40.581600 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.581585    2553 scope.go:117] "RemoveContainer" containerID="cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"
Jul 02 18:46:40.587535 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.587515    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:46:40.588610 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.588599    2553 scope.go:117] "RemoveContainer" containerID="45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"
Jul 02 18:46:40.592471 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.592451    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-predictor-6697b4c4cc-vn7wz"]
Jul 02 18:46:40.595723 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.595706    2553 scope.go:117] "RemoveContainer" containerID="510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"
Jul 02 18:46:40.596071 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:46:40.595961    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479\": container with ID starting with 510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479 not found: ID does not exist" containerID="510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"
Jul 02 18:46:40.596154 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596083    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479"} err="failed to get container status \"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479\": rpc error: code = NotFound desc = could not find container \"510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479\": container with ID starting with 510dcd5266cf7b7d0e03ff38d3f1a4377a706743008eff924f7588a277277479 not found: ID does not exist"
Jul 02 18:46:40.596154 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596107    2553 scope.go:117] "RemoveContainer" containerID="6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"
Jul 02 18:46:40.596357 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:46:40.596342    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0\": container with ID starting with 6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0 not found: ID does not exist" containerID="6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"
Jul 02 18:46:40.596416 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596364    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0"} err="failed to get container status \"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0\": rpc error: code = NotFound desc = could not find container \"6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0\": container with ID starting with 6d9a374fc081ad25621866ad3e25b371b85621095a614093e54280714bda52a0 not found: ID does not exist"
Jul 02 18:46:40.596416 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596378    2553 scope.go:117] "RemoveContainer" containerID="cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"
Jul 02 18:46:40.596603 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:46:40.596585    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a\": container with ID starting with cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a not found: ID does not exist" containerID="cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"
Jul 02 18:46:40.596664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596610    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a"} err="failed to get container status \"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a\": rpc error: code = NotFound desc = could not find container \"cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a\": container with ID starting with cd198aca34f37149f17a14aa340ae1be442cb18f5e80c1d09cdf01583b59b59a not found: ID does not exist"
Jul 02 18:46:40.596664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596630    2553 scope.go:117] "RemoveContainer" containerID="45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"
Jul 02 18:46:40.596840 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:46:40.596823    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a\": container with ID starting with 45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a not found: ID does not exist" containerID="45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"
Jul 02 18:46:40.596877 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:40.596843    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a"} err="failed to get container status \"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a\": rpc error: code = NotFound desc = could not find container \"45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a\": container with ID starting with 45c49d4c704bf2e5a804b73426fc2c36e736434b2501fc2c0df6c7f254614f8a not found: ID does not exist"
Jul 02 18:46:42.161378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:42.161348    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" path="/var/lib/kubelet/pods/1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c/volumes"
Jul 02 18:46:50.609391 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:50.609360    2553 generic.go:356] "Generic (PLEG): container finished" podID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerID="79dd951735875c919d0de276fd07099ee7afc39afce6d3209912855836434974" exitCode=0
Jul 02 18:46:50.609738 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:46:50.609436    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerDied","Data":"79dd951735875c919d0de276fd07099ee7afc39afce6d3209912855836434974"}
Jul 02 18:49:06.233058 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.233023    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerStarted","Data":"8779e4bc1bc0bc2bd344c371f70e55efd2346c181c5114eb38ce870090d87795"}
Jul 02 18:49:06.233565 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.233083    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:06.233565 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.233108    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:06.233565 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.233127    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerStarted","Data":"7862a81a262f019fbea9c09e52ed0b8d8152c9fbc74398629f0003161296b828"}
Jul 02 18:49:06.234408 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.234378    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:49:06.258665 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:06.258618    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podStartSLOduration=1.690965616 podStartE2EDuration="2m46.258604246s" totalImagesPullingTime="2m14.845324694s" totalInitContainerRuntime="29.722313936s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:49:06.256738231 +0000 UTC m=+1122.585987855" watchObservedRunningTime="2026-07-02 18:49:06.258604246 +0000 UTC m=+1122.587853873"
Jul 02 18:49:07.237544 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:07.237496    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:49:12.242427 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:12.242398    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:12.242986 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:12.242945    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.30:8080: connect: connection refused"
Jul 02 18:49:22.243718 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:22.243683    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:30.394859 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:30.394819    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:49:30.395378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:30.395197    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container" containerID="cri-o://7862a81a262f019fbea9c09e52ed0b8d8152c9fbc74398629f0003161296b828" gracePeriod=30
Jul 02 18:49:30.395378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:30.395263    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kube-rbac-proxy" containerID="cri-o://8779e4bc1bc0bc2bd344c371f70e55efd2346c181c5114eb38ce870090d87795" gracePeriod=30
Jul 02 18:49:31.340943 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.340910    2553 generic.go:356] "Generic (PLEG): container finished" podID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerID="8779e4bc1bc0bc2bd344c371f70e55efd2346c181c5114eb38ce870090d87795" exitCode=2
Jul 02 18:49:31.340943 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.340935    2553 generic.go:356] "Generic (PLEG): container finished" podID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerID="7862a81a262f019fbea9c09e52ed0b8d8152c9fbc74398629f0003161296b828" exitCode=0
Jul 02 18:49:31.341151 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.340966    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerDied","Data":"8779e4bc1bc0bc2bd344c371f70e55efd2346c181c5114eb38ce870090d87795"}
Jul 02 18:49:31.341151 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.341030    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerDied","Data":"7862a81a262f019fbea9c09e52ed0b8d8152c9fbc74398629f0003161296b828"}
Jul 02 18:49:31.378095 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.378052    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:31.517741 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.517644    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9krpk\" (UniqueName: \"kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk\") pod \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") "
Jul 02 18:49:31.517741 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.517712    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls\") pod \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") "
Jul 02 18:49:31.518306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.517758    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location\") pod \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") "
Jul 02 18:49:31.518306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.517782    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config\") pod \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") "
Jul 02 18:49:31.518306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.517809    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert\") pod \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\" (UID: \"74fc5a63-cf19-4b35-98e6-cba16ebd1db6\") "
Jul 02 18:49:31.518306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.518214    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "74fc5a63-cf19-4b35-98e6-cba16ebd1db6" (UID: "74fc5a63-cf19-4b35-98e6-cba16ebd1db6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:49:31.518306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.518251    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "74fc5a63-cf19-4b35-98e6-cba16ebd1db6" (UID: "74fc5a63-cf19-4b35-98e6-cba16ebd1db6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:49:31.519909 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.519880    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "74fc5a63-cf19-4b35-98e6-cba16ebd1db6" (UID: "74fc5a63-cf19-4b35-98e6-cba16ebd1db6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:49:31.520054 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.519905    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk" (OuterVolumeSpecName: "kube-api-access-9krpk") pod "74fc5a63-cf19-4b35-98e6-cba16ebd1db6" (UID: "74fc5a63-cf19-4b35-98e6-cba16ebd1db6"). InnerVolumeSpecName "kube-api-access-9krpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:49:31.522460 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.522431    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "74fc5a63-cf19-4b35-98e6-cba16ebd1db6" (UID: "74fc5a63-cf19-4b35-98e6-cba16ebd1db6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:49:31.618895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.618858    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:49:31.618895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.618888    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:49:31.618895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.618898    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:49:31.618895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.618906    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9krpk\" (UniqueName: \"kubernetes.io/projected/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-kube-api-access-9krpk\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:49:31.619189 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:31.618917    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/74fc5a63-cf19-4b35-98e6-cba16ebd1db6-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:49:32.347181 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.347128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6" event={"ID":"74fc5a63-cf19-4b35-98e6-cba16ebd1db6","Type":"ContainerDied","Data":"b7ef91b7de6eac877bbce201c67b2bb0f1cd90d5eaf67f052e7bad0a70a8ae96"}
Jul 02 18:49:32.347181 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.347176    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"
Jul 02 18:49:32.347378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.347255    2553 scope.go:117] "RemoveContainer" containerID="8779e4bc1bc0bc2bd344c371f70e55efd2346c181c5114eb38ce870090d87795"
Jul 02 18:49:32.355297 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.355270    2553 scope.go:117] "RemoveContainer" containerID="7862a81a262f019fbea9c09e52ed0b8d8152c9fbc74398629f0003161296b828"
Jul 02 18:49:32.362787 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.362765    2553 scope.go:117] "RemoveContainer" containerID="79dd951735875c919d0de276fd07099ee7afc39afce6d3209912855836434974"
Jul 02 18:49:32.365427 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.365405    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:49:32.369822 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:32.369798    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v1-predictor-7bbbbd79cd-rw6d6"]
Jul 02 18:49:34.161769 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:34.161737    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" path="/var/lib/kubelet/pods/74fc5a63-cf19-4b35-98e6-cba16ebd1db6/volumes"
Jul 02 18:49:35.709333 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709299    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709572    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709582    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709592    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709598    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709606    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="storage-initializer"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709612    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="storage-initializer"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709626    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="storage-initializer"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709631    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="storage-initializer"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709638    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709643    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709653    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kube-rbac-proxy"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709658    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kube-rbac-proxy"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709665    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709670    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709715    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kserve-container"
Jul 02 18:49:35.709716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709722    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kserve-container"
Jul 02 18:49:35.710252 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709728    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="74fc5a63-cf19-4b35-98e6-cba16ebd1db6" containerName="kube-rbac-proxy"
Jul 02 18:49:35.710252 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709736    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="agent"
Jul 02 18:49:35.710252 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.709742    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c1a5de8-ba6f-4af0-bdae-1ce0ed67354c" containerName="kube-rbac-proxy"
Jul 02 18:49:35.744472 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.744439    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:49:35.744642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.744564    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.747236 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747209    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-predictor-serving-cert\""
Jul 02 18:49:35.747236 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747228    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:49:35.747451 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747325    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:49:35.747451 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747334    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:49:35.747451 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747371    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2vv24\""
Jul 02 18:49:35.747600 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.747580    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-kube-rbac-proxy-sar-config\""
Jul 02 18:49:35.748522 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.748501    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:49:35.857695 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.857658    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.857892 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.857712    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwmz\" (UniqueName: \"kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.857892 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.857762    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.857892 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.857849    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.858088 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.857911    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958470 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.958432    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.958487    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.958520    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.958549    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958673 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.958573    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwmz\" (UniqueName: \"kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.958888 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:49:35.958707    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-predictor-serving-cert: secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 02 18:49:35.958888 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:49:35.958815    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls podName:867f90ed-50a1-4bd0-91a5-d3a7bc618bf0 nodeName:}" failed. No retries permitted until 2026-07-02 18:49:36.458791883 +0000 UTC m=+1152.788041487 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls") pod "isvc-autogluon-v2-predictor-67b849d456-nlv7z" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0") : secret "isvc-autogluon-v2-predictor-serving-cert" not found
Jul 02 18:49:35.959111 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.959073    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.959328 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.959311    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.959396 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.959317    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:35.967476 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:35.967453    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwmz\" (UniqueName: \"kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:36.463586 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:36.463549    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:36.466052 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:36.466025    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") pod \"isvc-autogluon-v2-predictor-67b849d456-nlv7z\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:36.655349 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:36.655306    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:49:36.805920 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:36.805878    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:49:36.809160 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:49:36.809130    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod867f90ed_50a1_4bd0_91a5_d3a7bc618bf0.slice/crio-8f4e796633296852838c3f21d06a8372774f1cfa3666ac2633663561b356b893 WatchSource:0}: Error finding container 8f4e796633296852838c3f21d06a8372774f1cfa3666ac2633663561b356b893: Status 404 returned error can't find the container with id 8f4e796633296852838c3f21d06a8372774f1cfa3666ac2633663561b356b893
Jul 02 18:49:37.372720 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:37.372680    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerStarted","Data":"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"}
Jul 02 18:49:37.372895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:49:37.372741    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerStarted","Data":"8f4e796633296852838c3f21d06a8372774f1cfa3666ac2633663561b356b893"}
Jul 02 18:50:07.492226 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:07.492188    2553 generic.go:356] "Generic (PLEG): container finished" podID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerID="419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc" exitCode=0
Jul 02 18:50:07.492746 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:07.492240    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerDied","Data":"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"}
Jul 02 18:50:08.498764 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:08.498726    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerStarted","Data":"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"}
Jul 02 18:50:08.499212 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:08.498791    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerStarted","Data":"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"}
Jul 02 18:50:08.499212 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:08.498998    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:50:08.518069 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:08.518018    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" podStartSLOduration=3.883778761 podStartE2EDuration="33.517999015s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.634220254s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:50:08.516696318 +0000 UTC m=+1184.845945940" watchObservedRunningTime="2026-07-02 18:50:08.517999015 +0000 UTC m=+1184.847248642"
Jul 02 18:50:09.503737 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:09.503707    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:50:15.512356 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:15.512327    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:50:24.109199 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:24.109167    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:50:24.112115 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:24.112095    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:50:24.116108 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:24.116088    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:50:24.119270 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:50:24.119252    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:51:43.160662 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:43.160584    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:51:45.853699 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:45.853667    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:51:45.854215 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:45.853917    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kube-rbac-proxy" containerID="cri-o://2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2" gracePeriod=30
Jul 02 18:51:45.854215 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:45.853907    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kserve-container" containerID="cri-o://a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b" gracePeriod=30
Jul 02 18:51:46.599530 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.599507    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:51:46.650707 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.650680    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config\") pod \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") "
Jul 02 18:51:46.650871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.650726    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location\") pod \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") "
Jul 02 18:51:46.650871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.650745    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert\") pod \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") "
Jul 02 18:51:46.650871 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.650862    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") pod \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") "
Jul 02 18:51:46.651036 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.650917    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrwmz\" (UniqueName: \"kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz\") pod \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\" (UID: \"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0\") "
Jul 02 18:51:46.651095 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.651074    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:51:46.651155 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.651089    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:51:46.651206 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.651172    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:51:46.651206 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.651187    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:51:46.652926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.652898    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:51:46.653120 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.652933    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz" (OuterVolumeSpecName: "kube-api-access-rrwmz") pod "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0"). InnerVolumeSpecName "kube-api-access-rrwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:51:46.655121 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.655093    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" (UID: "867f90ed-50a1-4bd0-91a5-d3a7bc618bf0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:51:46.751786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.751679    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:51:46.751786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.751727    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:51:46.751786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.751740    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwmz\" (UniqueName: \"kubernetes.io/projected/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0-kube-api-access-rrwmz\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:51:46.894064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894031    2553 generic.go:356] "Generic (PLEG): container finished" podID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerID="2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2" exitCode=2
Jul 02 18:51:46.894064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894056    2553 generic.go:356] "Generic (PLEG): container finished" podID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerID="a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b" exitCode=0
Jul 02 18:51:46.894520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894094    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"
Jul 02 18:51:46.894520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894109    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerDied","Data":"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"}
Jul 02 18:51:46.894520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894164    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerDied","Data":"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"}
Jul 02 18:51:46.894520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894185    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z" event={"ID":"867f90ed-50a1-4bd0-91a5-d3a7bc618bf0","Type":"ContainerDied","Data":"8f4e796633296852838c3f21d06a8372774f1cfa3666ac2633663561b356b893"}
Jul 02 18:51:46.894520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.894210    2553 scope.go:117] "RemoveContainer" containerID="2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"
Jul 02 18:51:46.905251 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.905231    2553 scope.go:117] "RemoveContainer" containerID="a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"
Jul 02 18:51:46.913292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.913273    2553 scope.go:117] "RemoveContainer" containerID="419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"
Jul 02 18:51:46.918358 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.918335    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:51:46.922306 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922287    2553 scope.go:117] "RemoveContainer" containerID="2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"
Jul 02 18:51:46.922527 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922510    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-predictor-67b849d456-nlv7z"]
Jul 02 18:51:46.922623 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:51:46.922587    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2\": container with ID starting with 2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2 not found: ID does not exist" containerID="2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"
Jul 02 18:51:46.922660 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922632    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"} err="failed to get container status \"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2\": rpc error: code = NotFound desc = could not find container \"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2\": container with ID starting with 2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2 not found: ID does not exist"
Jul 02 18:51:46.922660 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922652    2553 scope.go:117] "RemoveContainer" containerID="a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"
Jul 02 18:51:46.922909 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:51:46.922892    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b\": container with ID starting with a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b not found: ID does not exist" containerID="a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"
Jul 02 18:51:46.922947 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922915    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"} err="failed to get container status \"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b\": rpc error: code = NotFound desc = could not find container \"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b\": container with ID starting with a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b not found: ID does not exist"
Jul 02 18:51:46.922947 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.922931    2553 scope.go:117] "RemoveContainer" containerID="419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"
Jul 02 18:51:46.923217 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:51:46.923201    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc\": container with ID starting with 419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc not found: ID does not exist" containerID="419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"
Jul 02 18:51:46.923261 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923222    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"} err="failed to get container status \"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc\": rpc error: code = NotFound desc = could not find container \"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc\": container with ID starting with 419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc not found: ID does not exist"
Jul 02 18:51:46.923261 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923236    2553 scope.go:117] "RemoveContainer" containerID="2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"
Jul 02 18:51:46.923462 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923446    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2"} err="failed to get container status \"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2\": rpc error: code = NotFound desc = could not find container \"2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2\": container with ID starting with 2e897b90443bb2d9ea4aecaa847937bd0c766fcdbd056046924deaeb8ef5d5f2 not found: ID does not exist"
Jul 02 18:51:46.923510 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923463    2553 scope.go:117] "RemoveContainer" containerID="a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"
Jul 02 18:51:46.923665 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923651    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b"} err="failed to get container status \"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b\": rpc error: code = NotFound desc = could not find container \"a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b\": container with ID starting with a7b177408b270192afd0d55fd167a4398c3a11d90476a450b39325d6462fe22b not found: ID does not exist"
Jul 02 18:51:46.923716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923664    2553 scope.go:117] "RemoveContainer" containerID="419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"
Jul 02 18:51:46.923839 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:46.923822    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc"} err="failed to get container status \"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc\": rpc error: code = NotFound desc = could not find container \"419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc\": container with ID starting with 419582e52f967d8e0c06dd408934f9622512a9ae3a209f7ac472b63d7574e4dc not found: ID does not exist"
Jul 02 18:51:48.162124 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:48.162088    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" path="/var/lib/kubelet/pods/867f90ed-50a1-4bd0-91a5-d3a7bc618bf0/volumes"
Jul 02 18:51:51.025268 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025228    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025657    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="storage-initializer"
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025675    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="storage-initializer"
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025696    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kube-rbac-proxy"
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025704    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kube-rbac-proxy"
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025716    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kserve-container"
Jul 02 18:51:51.025756 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025724    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kserve-container"
Jul 02 18:51:51.026092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025792    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kube-rbac-proxy"
Jul 02 18:51:51.026092 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.025805    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="867f90ed-50a1-4bd0-91a5-d3a7bc618bf0" containerName="kserve-container"
Jul 02 18:51:51.029181 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.029161    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.031894 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.031868    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:51:51.032130 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.032115    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 02 18:51:51.032214 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.032130    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2vv24\""
Jul 02 18:51:51.032214 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.032189    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:51:51.033136 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.033118    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:51:51.033218 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.033145    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 02 18:51:51.033275 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.033231    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:51:51.038501 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.038475    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:51:51.086786 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.086747    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.086967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.086808    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.086967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.086870    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.086967 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.086946    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.087093 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.087009    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.187949 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.187917    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188093 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.187967    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188093 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188006    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188093 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188044    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188093 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188076    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188298 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:51:51.188197    2553 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-serving-cert: secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 02 18:51:51.188298 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:51:51.188285    2553 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls podName:25ea193a-a512-4625-86f5-ef0041ea2852 nodeName:}" failed. No retries permitted until 2026-07-02 18:51:51.688265479 +0000 UTC m=+1288.017515086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls") pod "isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852") : secret "isvc-autogluon-v2-variants-predictor-serving-cert" not found
Jul 02 18:51:51.188463 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188439    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188652 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188633    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.188652 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.188646    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.198810 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.198779    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.691927 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.691876    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.694314 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.694285    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:51.939926 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:51.939887    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:51:52.067344 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:52.067311    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:51:52.070727 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:51:52.070700    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ea193a_a512_4625_86f5_ef0041ea2852.slice/crio-33de0383914990a78a53075b237de7f3a4f9ad20439ed8ebc57e67ead5fc2643 WatchSource:0}: Error finding container 33de0383914990a78a53075b237de7f3a4f9ad20439ed8ebc57e67ead5fc2643: Status 404 returned error can't find the container with id 33de0383914990a78a53075b237de7f3a4f9ad20439ed8ebc57e67ead5fc2643
Jul 02 18:51:52.926799 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:52.926760    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerStarted","Data":"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"}
Jul 02 18:51:52.927010 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:51:52.926832    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerStarted","Data":"33de0383914990a78a53075b237de7f3a4f9ad20439ed8ebc57e67ead5fc2643"}
Jul 02 18:52:13.016815 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:13.016721    2553 generic.go:356] "Generic (PLEG): container finished" podID="25ea193a-a512-4625-86f5-ef0041ea2852" containerID="fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76" exitCode=0
Jul 02 18:52:13.016815 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:13.016795    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerDied","Data":"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"}
Jul 02 18:52:14.023516 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:14.023483    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerStarted","Data":"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"}
Jul 02 18:52:14.024064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:14.023544    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerStarted","Data":"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"}
Jul 02 18:52:14.024064 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:14.023734    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:52:14.042410 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:14.042364    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" podStartSLOduration=2.400098717 podStartE2EDuration="23.042347093s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.642248376s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:52:14.040377538 +0000 UTC m=+1310.369627186" watchObservedRunningTime="2026-07-02 18:52:14.042347093 +0000 UTC m=+1310.371596716"
Jul 02 18:52:15.028677 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:15.028643    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:52:21.040046 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:52:21.040017    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:53:48.162528 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:48.162499    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:53:51.172308 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.172276    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:53:51.172727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.172537    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kserve-container" containerID="cri-o://9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344" gracePeriod=30
Jul 02 18:53:51.172727 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.172547    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kube-rbac-proxy" containerID="cri-o://18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900" gracePeriod=30
Jul 02 18:53:51.410772 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.410738    2553 generic.go:356] "Generic (PLEG): container finished" podID="25ea193a-a512-4625-86f5-ef0041ea2852" containerID="18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900" exitCode=2
Jul 02 18:53:51.410944 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.410820    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerDied","Data":"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"}
Jul 02 18:53:51.913208 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:51.913184    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:53:52.062710 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.062638    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location\") pod \"25ea193a-a512-4625-86f5-ef0041ea2852\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") "
Jul 02 18:53:52.062710 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.062674    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config\") pod \"25ea193a-a512-4625-86f5-ef0041ea2852\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") "
Jul 02 18:53:52.062921 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.062716    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert\") pod \"25ea193a-a512-4625-86f5-ef0041ea2852\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") "
Jul 02 18:53:52.062921 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.062751    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg\") pod \"25ea193a-a512-4625-86f5-ef0041ea2852\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") "
Jul 02 18:53:52.062921 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.062787    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") pod \"25ea193a-a512-4625-86f5-ef0041ea2852\" (UID: \"25ea193a-a512-4625-86f5-ef0041ea2852\") "
Jul 02 18:53:52.063110 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.063091    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25ea193a-a512-4625-86f5-ef0041ea2852" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:53:52.063167 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.063138    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25ea193a-a512-4625-86f5-ef0041ea2852" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:53:52.064879 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.064855    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25ea193a-a512-4625-86f5-ef0041ea2852" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:53:52.064879 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.064867    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg" (OuterVolumeSpecName: "kube-api-access-vfvlg") pod "25ea193a-a512-4625-86f5-ef0041ea2852" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852"). InnerVolumeSpecName "kube-api-access-vfvlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:53:52.067323 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.067298    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25ea193a-a512-4625-86f5-ef0041ea2852" (UID: "25ea193a-a512-4625-86f5-ef0041ea2852"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:53:52.163378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.163349    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25ea193a-a512-4625-86f5-ef0041ea2852-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:53:52.163378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.163373    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:53:52.163560 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.163387    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25ea193a-a512-4625-86f5-ef0041ea2852-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:53:52.163560 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.163400    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvlg\" (UniqueName: \"kubernetes.io/projected/25ea193a-a512-4625-86f5-ef0041ea2852-kube-api-access-vfvlg\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:53:52.163560 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.163414    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25ea193a-a512-4625-86f5-ef0041ea2852-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:53:52.416750 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.416717    2553 generic.go:356] "Generic (PLEG): container finished" podID="25ea193a-a512-4625-86f5-ef0041ea2852" containerID="9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344" exitCode=0
Jul 02 18:53:52.417156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.416775    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"
Jul 02 18:53:52.417156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.416772    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerDied","Data":"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"}
Jul 02 18:53:52.417156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.416901    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk" event={"ID":"25ea193a-a512-4625-86f5-ef0041ea2852","Type":"ContainerDied","Data":"33de0383914990a78a53075b237de7f3a4f9ad20439ed8ebc57e67ead5fc2643"}
Jul 02 18:53:52.417156 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.416943    2553 scope.go:117] "RemoveContainer" containerID="18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"
Jul 02 18:53:52.424810 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.424795    2553 scope.go:117] "RemoveContainer" containerID="9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"
Jul 02 18:53:52.431990 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.431960    2553 scope.go:117] "RemoveContainer" containerID="fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"
Jul 02 18:53:52.434574 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.434555    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:53:52.438872 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.438847    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-c6767c94c-rfrdk"]
Jul 02 18:53:52.440173 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440158    2553 scope.go:117] "RemoveContainer" containerID="18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"
Jul 02 18:53:52.440404 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:53:52.440387    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900\": container with ID starting with 18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900 not found: ID does not exist" containerID="18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"
Jul 02 18:53:52.440459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440410    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900"} err="failed to get container status \"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900\": rpc error: code = NotFound desc = could not find container \"18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900\": container with ID starting with 18a738fafa6359a1a39ce298089580f306aa80bb71ede52db1b04671c4ada900 not found: ID does not exist"
Jul 02 18:53:52.440459 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440425    2553 scope.go:117] "RemoveContainer" containerID="9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"
Jul 02 18:53:52.440639 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:53:52.440621    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344\": container with ID starting with 9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344 not found: ID does not exist" containerID="9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"
Jul 02 18:53:52.440677 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440651    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344"} err="failed to get container status \"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344\": rpc error: code = NotFound desc = could not find container \"9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344\": container with ID starting with 9a922fbe90ebb49f41a08cf2ffa4b43039f852f0068ff813caaa1ccce07af344 not found: ID does not exist"
Jul 02 18:53:52.440677 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440668    2553 scope.go:117] "RemoveContainer" containerID="fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"
Jul 02 18:53:52.440910 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:53:52.440892    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76\": container with ID starting with fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76 not found: ID does not exist" containerID="fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"
Jul 02 18:53:52.440968 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:52.440915    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76"} err="failed to get container status \"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76\": rpc error: code = NotFound desc = could not find container \"fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76\": container with ID starting with fa18f1265e82d7f43d6aee158df3e78f1d9e4947914abc1f0198c322d084fc76 not found: ID does not exist"
Jul 02 18:53:54.161678 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:54.161648    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" path="/var/lib/kubelet/pods/25ea193a-a512-4625-86f5-ef0041ea2852/volumes"
Jul 02 18:53:56.331008 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.330957    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331275    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kube-rbac-proxy"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331285    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kube-rbac-proxy"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331294    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="storage-initializer"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331300    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="storage-initializer"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331315    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kserve-container"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331320    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kserve-container"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331364    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kserve-container"
Jul 02 18:53:56.331399 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.331372    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ea193a-a512-4625-86f5-ef0041ea2852" containerName="kube-rbac-proxy"
Jul 02 18:53:56.334525 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.334493    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.337259 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.337234    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2vv24\""
Jul 02 18:53:56.337385 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.337244    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:53:56.337385 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.337270    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:53:56.340678 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.340652    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-predictor-serving-cert\""
Jul 02 18:53:56.340814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.340652    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:53:56.340814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.340719    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:53:56.340814 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.340795    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-noslash-kube-rbac-proxy-sar-config\""
Jul 02 18:53:56.345191 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.345168    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:53:56.397305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.397264    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.397475 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.397321    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.397475 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.397369    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.397475 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.397415    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.397475 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.397449    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98xb\" (UniqueName: \"kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.498655 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.498614    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.498655 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.498656    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.498935 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.498688    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.498935 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.498715    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98xb\" (UniqueName: \"kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.498935 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.498747    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.499155 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.499126    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.499413 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.499394    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.499490 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.499469    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.501210 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.501192    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.506568 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.506550    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98xb\" (UniqueName: \"kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb\") pod \"isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.644753 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.644664    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:53:56.779101 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:56.779075    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:53:56.781636 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:53:56.781606    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b3938d_cc1f_4d6e_ab0f_2edc17d5e835.slice/crio-aa23d24e7ce1e8566d3dffb810c175a451d7c42e0e23efc077d46d60f0d6d9ef WatchSource:0}: Error finding container aa23d24e7ce1e8566d3dffb810c175a451d7c42e0e23efc077d46d60f0d6d9ef: Status 404 returned error can't find the container with id aa23d24e7ce1e8566d3dffb810c175a451d7c42e0e23efc077d46d60f0d6d9ef
Jul 02 18:53:57.439358 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:57.439323    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerStarted","Data":"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"}
Jul 02 18:53:57.439753 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:53:57.439385    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerStarted","Data":"aa23d24e7ce1e8566d3dffb810c175a451d7c42e0e23efc077d46d60f0d6d9ef"}
Jul 02 18:54:01.458797 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:01.458764    2553 generic.go:356] "Generic (PLEG): container finished" podID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerID="3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22" exitCode=0
Jul 02 18:54:01.459202 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:01.458844    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerDied","Data":"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"}
Jul 02 18:54:02.464762 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:02.464720    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerStarted","Data":"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"}
Jul 02 18:54:02.465199 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:02.464806    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerStarted","Data":"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"}
Jul 02 18:54:02.465199 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:02.465037    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:54:02.482669 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:02.482603    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" podStartSLOduration=2.039541505 podStartE2EDuration="6.482582811s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.443041306s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:54:02.482184854 +0000 UTC m=+1418.811434476" watchObservedRunningTime="2026-07-02 18:54:02.482582811 +0000 UTC m=+1418.811832434"
Jul 02 18:54:03.469438 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:03.469400    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:54:09.479780 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:54:09.479748    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:55:24.135738 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:24.135704    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:55:24.139724 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:24.139703    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:55:24.144293 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:24.144269    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:55:24.147706 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:24.147689    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:55:35.160099 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:35.160070    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:55:36.470264 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:36.470229    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:55:36.851111 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:36.851015    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kserve-container" containerID="cri-o://085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f" gracePeriod=30
Jul 02 18:55:36.851292 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:36.851090    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kube-rbac-proxy" containerID="cri-o://0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c" gracePeriod=30
Jul 02 18:55:37.695803 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.695779    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:55:37.713160 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713134    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config\") pod \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") "
Jul 02 18:55:37.713308 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713175    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z98xb\" (UniqueName: \"kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb\") pod \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") "
Jul 02 18:55:37.713308 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713200    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location\") pod \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") "
Jul 02 18:55:37.713308 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713245    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert\") pod \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") "
Jul 02 18:55:37.713308 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713279    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls\") pod \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\" (UID: \"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835\") "
Jul 02 18:55:37.713531 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713467    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" (UID: "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:55:37.713580 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.713552    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" (UID: "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:55:37.715591 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.715563    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb" (OuterVolumeSpecName: "kube-api-access-z98xb") pod "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" (UID: "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835"). InnerVolumeSpecName "kube-api-access-z98xb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:55:37.715705 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.715681    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" (UID: "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:55:37.719051 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.719024    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" (UID: "37b3938d-cc1f-4d6e-ab0f-2edc17d5e835"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:55:37.814083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.813960    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:55:37.814083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.814019    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:55:37.814083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.814032    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:55:37.814083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.814044    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z98xb\" (UniqueName: \"kubernetes.io/projected/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kube-api-access-z98xb\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:55:37.814083 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.814056    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:55:37.856604 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856568    2553 generic.go:356] "Generic (PLEG): container finished" podID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerID="0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c" exitCode=2
Jul 02 18:55:37.856604 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856601    2553 generic.go:356] "Generic (PLEG): container finished" podID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerID="085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f" exitCode=0
Jul 02 18:55:37.856823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856633    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"
Jul 02 18:55:37.856823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856642    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerDied","Data":"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"}
Jul 02 18:55:37.856823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856705    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerDied","Data":"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"}
Jul 02 18:55:37.856823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856738    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77" event={"ID":"37b3938d-cc1f-4d6e-ab0f-2edc17d5e835","Type":"ContainerDied","Data":"aa23d24e7ce1e8566d3dffb810c175a451d7c42e0e23efc077d46d60f0d6d9ef"}
Jul 02 18:55:37.856823 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.856787    2553 scope.go:117] "RemoveContainer" containerID="0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"
Jul 02 18:55:37.866254 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.866236    2553 scope.go:117] "RemoveContainer" containerID="085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"
Jul 02 18:55:37.873404 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.873383    2553 scope.go:117] "RemoveContainer" containerID="3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"
Jul 02 18:55:37.879067 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.879044    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:55:37.880960 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.880942    2553 scope.go:117] "RemoveContainer" containerID="0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"
Jul 02 18:55:37.881242 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:55:37.881223    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c\": container with ID starting with 0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c not found: ID does not exist" containerID="0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"
Jul 02 18:55:37.881336 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881252    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"} err="failed to get container status \"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c\": rpc error: code = NotFound desc = could not find container \"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c\": container with ID starting with 0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c not found: ID does not exist"
Jul 02 18:55:37.881336 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881269    2553 scope.go:117] "RemoveContainer" containerID="085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"
Jul 02 18:55:37.881521 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:55:37.881502    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f\": container with ID starting with 085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f not found: ID does not exist" containerID="085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"
Jul 02 18:55:37.881596 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881526    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"} err="failed to get container status \"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f\": rpc error: code = NotFound desc = could not find container \"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f\": container with ID starting with 085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f not found: ID does not exist"
Jul 02 18:55:37.881596 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881543    2553 scope.go:117] "RemoveContainer" containerID="3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"
Jul 02 18:55:37.881845 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:55:37.881827    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22\": container with ID starting with 3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22 not found: ID does not exist" containerID="3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"
Jul 02 18:55:37.881921 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881851    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"} err="failed to get container status \"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22\": rpc error: code = NotFound desc = could not find container \"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22\": container with ID starting with 3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22 not found: ID does not exist"
Jul 02 18:55:37.881921 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.881868    2553 scope.go:117] "RemoveContainer" containerID="0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"
Jul 02 18:55:37.882142 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882111    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c"} err="failed to get container status \"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c\": rpc error: code = NotFound desc = could not find container \"0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c\": container with ID starting with 0a5f28888d121954d7163671b9d4d8622cf4600902902c2595d0786bf031a14c not found: ID does not exist"
Jul 02 18:55:37.882209 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882144    2553 scope.go:117] "RemoveContainer" containerID="085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"
Jul 02 18:55:37.882383 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882362    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f"} err="failed to get container status \"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f\": rpc error: code = NotFound desc = could not find container \"085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f\": container with ID starting with 085fd876786d67ef34f4311d5bb210ff2ec3cd367c1afe24dba1786d6c8c2e5f not found: ID does not exist"
Jul 02 18:55:37.882448 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882384    2553 scope.go:117] "RemoveContainer" containerID="3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"
Jul 02 18:55:37.882585 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882566    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-noslash-predictor-9dcfbbb84-zmq77"]
Jul 02 18:55:37.882636 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:37.882584    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22"} err="failed to get container status \"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22\": rpc error: code = NotFound desc = could not find container \"3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22\": container with ID starting with 3ba8479cb7c83deb1a23a836821042576355d3a51ffc496011dca9205b7b8a22 not found: ID does not exist"
Jul 02 18:55:38.161548 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:38.161522    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" path="/var/lib/kubelet/pods/37b3938d-cc1f-4d6e-ab0f-2edc17d5e835/volumes"
Jul 02 18:55:41.631565 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.631523    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.631995    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="storage-initializer"
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632015    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="storage-initializer"
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632036    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kube-rbac-proxy"
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632045    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kube-rbac-proxy"
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632062    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kserve-container"
Jul 02 18:55:41.632078 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632070    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kserve-container"
Jul 02 18:55:41.632402 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632156    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kserve-container"
Jul 02 18:55:41.632402 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.632167    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="37b3938d-cc1f-4d6e-ab0f-2edc17d5e835" containerName="kube-rbac-proxy"
Jul 02 18:55:41.636952 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.636928    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.641050 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641030    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-2vv24\""
Jul 02 18:55:41.641155 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641070    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 02 18:55:41.641318 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641297    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 02 18:55:41.641869 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641851    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-predictor-serving-cert\""
Jul 02 18:55:41.642042 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641893    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-kube-rbac-proxy-sar-config\""
Jul 02 18:55:41.642042 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641916    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 02 18:55:41.642042 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.641935    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 02 18:55:41.650566 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.650543    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:55:41.740697 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.740660    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6vcj\" (UniqueName: \"kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.740697 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.740703    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.740964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.740775    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.740964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.740798    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.740964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.740815    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841492 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841461    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841501    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841545    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841573    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6vcj\" (UniqueName: \"kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841642 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841595    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.841964 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.841940    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.842259 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.842234    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.842369 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.842262    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.844010 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.843968    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.849458 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.849437    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6vcj\" (UniqueName: \"kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj\") pod \"isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:41.946644 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:41.946611    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:42.074762 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:42.074739    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:55:42.076693 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:55:42.076664    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6204689b_9784_48ea_a45f_55ca43a2476e.slice/crio-da38d3e95614a7d1499da1378715effd66dff95b1b456a701314529fea903689 WatchSource:0}: Error finding container da38d3e95614a7d1499da1378715effd66dff95b1b456a701314529fea903689: Status 404 returned error can't find the container with id da38d3e95614a7d1499da1378715effd66dff95b1b456a701314529fea903689
Jul 02 18:55:42.881298 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:42.881262    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerStarted","Data":"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"}
Jul 02 18:55:42.881687 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:42.881324    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerStarted","Data":"da38d3e95614a7d1499da1378715effd66dff95b1b456a701314529fea903689"}
Jul 02 18:55:46.897909 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:46.897814    2553 generic.go:356] "Generic (PLEG): container finished" podID="6204689b-9784-48ea-a45f-55ca43a2476e" containerID="64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0" exitCode=0
Jul 02 18:55:46.897909 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:46.897887    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerDied","Data":"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"}
Jul 02 18:55:47.905593 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:47.905561    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerStarted","Data":"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"}
Jul 02 18:55:47.905996 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:47.905618    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerStarted","Data":"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"}
Jul 02 18:55:47.905996 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:47.905813    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:47.925672 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:47.925611    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podStartSLOduration=2.5109501400000003 podStartE2EDuration="6.925590075s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.414639935s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:55:47.923848239 +0000 UTC m=+1524.253097885" watchObservedRunningTime="2026-07-02 18:55:47.925590075 +0000 UTC m=+1524.254839697"
Jul 02 18:55:48.909875 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:48.909838    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:48.911074 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:48.911040    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 18:55:49.914019 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:49.913961    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 18:55:54.918016 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:54.917987    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:55:54.918509 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:55:54.918481    2553 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.132.0.34:8080: connect: connection refused"
Jul 02 18:56:04.919929 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:04.919898    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:56:11.688168 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:11.688137    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:56:11.688728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:11.688378    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container" containerID="cri-o://08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4" gracePeriod=30
Jul 02 18:56:11.688728 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:11.688404    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kube-rbac-proxy" containerID="cri-o://d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859" gracePeriod=30
Jul 02 18:56:12.011536 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.011447    2553 generic.go:356] "Generic (PLEG): container finished" podID="6204689b-9784-48ea-a45f-55ca43a2476e" containerID="d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859" exitCode=2
Jul 02 18:56:12.011536 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.011496    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerDied","Data":"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"}
Jul 02 18:56:12.539203 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.539182    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:56:12.687961 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.687928    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t6vcj\" (UniqueName: \"kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj\") pod \"6204689b-9784-48ea-a45f-55ca43a2476e\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") "
Jul 02 18:56:12.688161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.687968    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert\") pod \"6204689b-9784-48ea-a45f-55ca43a2476e\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") "
Jul 02 18:56:12.688161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.688023    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location\") pod \"6204689b-9784-48ea-a45f-55ca43a2476e\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") "
Jul 02 18:56:12.688161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.688058    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls\") pod \"6204689b-9784-48ea-a45f-55ca43a2476e\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") "
Jul 02 18:56:12.688161 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.688086    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config\") pod \"6204689b-9784-48ea-a45f-55ca43a2476e\" (UID: \"6204689b-9784-48ea-a45f-55ca43a2476e\") "
Jul 02 18:56:12.688632 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.688445    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6204689b-9784-48ea-a45f-55ca43a2476e" (UID: "6204689b-9784-48ea-a45f-55ca43a2476e"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:56:12.688632 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.688495    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6204689b-9784-48ea-a45f-55ca43a2476e" (UID: "6204689b-9784-48ea-a45f-55ca43a2476e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 18:56:12.689107 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.689085    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6204689b-9784-48ea-a45f-55ca43a2476e" (UID: "6204689b-9784-48ea-a45f-55ca43a2476e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:56:12.690095 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.690071    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6204689b-9784-48ea-a45f-55ca43a2476e" (UID: "6204689b-9784-48ea-a45f-55ca43a2476e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 18:56:12.690191 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.690121    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj" (OuterVolumeSpecName: "kube-api-access-t6vcj") pod "6204689b-9784-48ea-a45f-55ca43a2476e" (UID: "6204689b-9784-48ea-a45f-55ca43a2476e"). InnerVolumeSpecName "kube-api-access-t6vcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:56:12.788545 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.788521    2553 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:12.788545 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.788541    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6vcj\" (UniqueName: \"kubernetes.io/projected/6204689b-9784-48ea-a45f-55ca43a2476e-kube-api-access-t6vcj\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:12.788694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.788551    2553 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6204689b-9784-48ea-a45f-55ca43a2476e-cabundle-cert\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:12.788694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.788559    2553 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6204689b-9784-48ea-a45f-55ca43a2476e-kserve-provision-location\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:12.788694 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:12.788567    2553 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6204689b-9784-48ea-a45f-55ca43a2476e-proxy-tls\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:13.017543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.017457    2553 generic.go:356] "Generic (PLEG): container finished" podID="6204689b-9784-48ea-a45f-55ca43a2476e" containerID="08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4" exitCode=0
Jul 02 18:56:13.017543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.017500    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerDied","Data":"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"}
Jul 02 18:56:13.017543 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.017517    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"
Jul 02 18:56:13.017804 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.017559    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx" event={"ID":"6204689b-9784-48ea-a45f-55ca43a2476e","Type":"ContainerDied","Data":"da38d3e95614a7d1499da1378715effd66dff95b1b456a701314529fea903689"}
Jul 02 18:56:13.017804 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.017587    2553 scope.go:117] "RemoveContainer" containerID="d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"
Jul 02 18:56:13.027287 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.027270    2553 scope.go:117] "RemoveContainer" containerID="08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"
Jul 02 18:56:13.034732 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.034711    2553 scope.go:117] "RemoveContainer" containerID="64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"
Jul 02 18:56:13.040848 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.040815    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:56:13.045159 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.045135    2553 scope.go:117] "RemoveContainer" containerID="d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"
Jul 02 18:56:13.046694 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:13.046670    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859\": container with ID starting with d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859 not found: ID does not exist" containerID="d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"
Jul 02 18:56:13.046803 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.046704    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859"} err="failed to get container status \"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859\": rpc error: code = NotFound desc = could not find container \"d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859\": container with ID starting with d0aa9e083d9a9dd22c3e7cfc9b13e9b65de00502560c3c0d191a32d88d6ba859 not found: ID does not exist"
Jul 02 18:56:13.046803 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.046723    2553 scope.go:117] "RemoveContainer" containerID="08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"
Jul 02 18:56:13.047126 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:13.047106    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4\": container with ID starting with 08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4 not found: ID does not exist" containerID="08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"
Jul 02 18:56:13.047193 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.047131    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4"} err="failed to get container status \"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4\": rpc error: code = NotFound desc = could not find container \"08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4\": container with ID starting with 08d7946c392116db61ca5260d690738279931772599b29f4ea9a219cc7fda0f4 not found: ID does not exist"
Jul 02 18:56:13.047193 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.047146    2553 scope.go:117] "RemoveContainer" containerID="64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"
Jul 02 18:56:13.047428 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:13.047410    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0\": container with ID starting with 64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0 not found: ID does not exist" containerID="64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"
Jul 02 18:56:13.047488 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.047437    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0"} err="failed to get container status \"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0\": rpc error: code = NotFound desc = could not find container \"64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0\": container with ID starting with 64f83b59a49d3a4e473daa936a3b05ae8353d94c8ea3fcc09aeb9fd2082cffc0 not found: ID does not exist"
Jul 02 18:56:13.047680 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:13.047664    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-predictor-89d67dbf7-pnkxx"]
Jul 02 18:56:14.161604 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:14.161574    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" path="/var/lib/kubelet/pods/6204689b-9784-48ea-a45f-55ca43a2476e/volumes"
Jul 02 18:56:18.269755 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.269724    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-sptcl/must-gather-z8lxg"]
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270012    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="storage-initializer"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270024    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="storage-initializer"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270039    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270045    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270057    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kube-rbac-proxy"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270062    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kube-rbac-proxy"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270108    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kserve-container"
Jul 02 18:56:18.270176 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.270119    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="6204689b-9784-48ea-a45f-55ca43a2476e" containerName="kube-rbac-proxy"
Jul 02 18:56:18.274895 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.274865    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.277896 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.277874    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-sptcl\"/\"default-dockercfg-mcssd\""
Jul 02 18:56:18.278247 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.278230    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sptcl\"/\"openshift-service-ca.crt\""
Jul 02 18:56:18.278357 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.278309    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-sptcl\"/\"kube-root-ca.crt\""
Jul 02 18:56:18.285143 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.285123    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sptcl/must-gather-z8lxg"]
Jul 02 18:56:18.429323 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.429298    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.429475 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.429329    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8x7l\" (UniqueName: \"kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.529767 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.529691    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.529767 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.529734    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8x7l\" (UniqueName: \"kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.530036 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.530018    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.538210 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.538191    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8x7l\" (UniqueName: \"kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l\") pod \"must-gather-z8lxg\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") " pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.584242 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.584221    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:18.708446 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.708309    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-sptcl/must-gather-z8lxg"]
Jul 02 18:56:18.711167 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:56:18.711140    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0e4e065d_614d_4115_9ece_90f87b70cacb.slice/crio-886295cfb40b148f281f1e54f7e6470c0b9f1636344adfdcb643516cfa484242 WatchSource:0}: Error finding container 886295cfb40b148f281f1e54f7e6470c0b9f1636344adfdcb643516cfa484242: Status 404 returned error can't find the container with id 886295cfb40b148f281f1e54f7e6470c0b9f1636344adfdcb643516cfa484242
Jul 02 18:56:18.713108 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:18.713088    2553 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 18:56:19.042030 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:19.041999    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sptcl/must-gather-z8lxg" event={"ID":"0e4e065d-614d-4115-9ece-90f87b70cacb","Type":"ContainerStarted","Data":"886295cfb40b148f281f1e54f7e6470c0b9f1636344adfdcb643516cfa484242"}
Jul 02 18:56:25.081114 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:25.081072    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sptcl/must-gather-z8lxg" event={"ID":"0e4e065d-614d-4115-9ece-90f87b70cacb","Type":"ContainerStarted","Data":"31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b"}
Jul 02 18:56:25.081502 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:25.081128    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sptcl/must-gather-z8lxg" event={"ID":"0e4e065d-614d-4115-9ece-90f87b70cacb","Type":"ContainerStarted","Data":"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"}
Jul 02 18:56:25.099879 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:25.099810    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-sptcl/must-gather-z8lxg" podStartSLOduration=1.170296594 podStartE2EDuration="7.099789289s" totalImagesPullingTime="5.929492695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 18:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:56:25.097421618 +0000 UTC m=+1561.426671243" watchObservedRunningTime="2026-07-02 18:56:25.099789289 +0000 UTC m=+1561.429038913"
Jul 02 18:56:45.169935 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:45.169894    2553 generic.go:356] "Generic (PLEG): container finished" podID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerID="686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f" exitCode=0
Jul 02 18:56:45.170378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:45.169967    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-sptcl/must-gather-z8lxg" event={"ID":"0e4e065d-614d-4115-9ece-90f87b70cacb","Type":"ContainerDied","Data":"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"}
Jul 02 18:56:45.170378 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:45.170317    2553 scope.go:117] "RemoveContainer" containerID="686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"
Jul 02 18:56:45.444237 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:45.444158    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sptcl_must-gather-z8lxg_0e4e065d-614d-4115-9ece-90f87b70cacb/gather/0.log"
Jul 02 18:56:48.388363 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:48.388325    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-d5rdt_07a2791d-f19a-4d3a-9a53-1ec166771a43/global-pull-secret-syncer/0.log"
Jul 02 18:56:48.508700 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:48.508669    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-f6qtv_39829f56-6d6f-41cf-9d77-cc91777211e4/konnectivity-agent/0.log"
Jul 02 18:56:48.563229 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:48.563201    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-27.ec2.internal_ab6471031b369a203d6efad60cbe0235/haproxy/0.log"
Jul 02 18:56:50.834250 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:50.834207    2553 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-sptcl/must-gather-z8lxg"]
Jul 02 18:56:50.834752 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:50.834435    2553 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-must-gather-sptcl/must-gather-z8lxg" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="copy" containerID="cri-o://31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b" gracePeriod=2
Jul 02 18:56:50.839374 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:50.839337    2553 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-sptcl/must-gather-z8lxg"]
Jul 02 18:56:51.070759 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.070736    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sptcl_must-gather-z8lxg_0e4e065d-614d-4115-9ece-90f87b70cacb/copy/0.log"
Jul 02 18:56:51.072321 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.072303    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:51.074746 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:51.074723    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8lxg\" is forbidden: User \"system:node:ip-10-0-130-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sptcl\": no relationship found between node 'ip-10-0-130-27.ec2.internal' and this object" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:51.097764 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.097702    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w8x7l\" (UniqueName: \"kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l\") pod \"0e4e065d-614d-4115-9ece-90f87b70cacb\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") "
Jul 02 18:56:51.097764 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.097759    2553 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output\") pod \"0e4e065d-614d-4115-9ece-90f87b70cacb\" (UID: \"0e4e065d-614d-4115-9ece-90f87b70cacb\") "
Jul 02 18:56:51.099128 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.099099    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "0e4e065d-614d-4115-9ece-90f87b70cacb" (UID: "0e4e065d-614d-4115-9ece-90f87b70cacb"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 18:56:51.099893 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.099865    2553 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l" (OuterVolumeSpecName: "kube-api-access-w8x7l") pod "0e4e065d-614d-4115-9ece-90f87b70cacb" (UID: "0e4e065d-614d-4115-9ece-90f87b70cacb"). InnerVolumeSpecName "kube-api-access-w8x7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 18:56:51.194309 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.194284    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-sptcl_must-gather-z8lxg_0e4e065d-614d-4115-9ece-90f87b70cacb/copy/0.log"
Jul 02 18:56:51.194664 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.194639    2553 generic.go:356] "Generic (PLEG): container finished" podID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerID="31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b" exitCode=143
Jul 02 18:56:51.194785 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.194688    2553 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:51.194785 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.194733    2553 scope.go:117] "RemoveContainer" containerID="31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b"
Jul 02 18:56:51.197498 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:51.197466    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8lxg\" is forbidden: User \"system:node:ip-10-0-130-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sptcl\": no relationship found between node 'ip-10-0-130-27.ec2.internal' and this object" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:51.198679 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.198657    2553 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8x7l\" (UniqueName: \"kubernetes.io/projected/0e4e065d-614d-4115-9ece-90f87b70cacb-kube-api-access-w8x7l\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:51.198679 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.198679    2553 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/0e4e065d-614d-4115-9ece-90f87b70cacb-must-gather-output\") on node \"ip-10-0-130-27.ec2.internal\" DevicePath \"\""
Jul 02 18:56:51.202573 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.202397    2553 scope.go:117] "RemoveContainer" containerID="686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"
Jul 02 18:56:51.205458 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:51.205433    2553 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-z8lxg\" is forbidden: User \"system:node:ip-10-0-130-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-sptcl\": no relationship found between node 'ip-10-0-130-27.ec2.internal' and this object" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" pod="openshift-must-gather-sptcl/must-gather-z8lxg"
Jul 02 18:56:51.214901 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.214881    2553 scope.go:117] "RemoveContainer" containerID="31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b"
Jul 02 18:56:51.215213 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:51.215192    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b\": container with ID starting with 31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b not found: ID does not exist" containerID="31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b"
Jul 02 18:56:51.215262 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.215223    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b"} err="failed to get container status \"31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b\": rpc error: code = NotFound desc = could not find container \"31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b\": container with ID starting with 31c86b18af3ae435dd719b73379dc5924a80038c9e2c36ebc5484d72d32a029b not found: ID does not exist"
Jul 02 18:56:51.215262 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.215240    2553 scope.go:117] "RemoveContainer" containerID="686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"
Jul 02 18:56:51.215469 ip-10-0-130-27 kubenswrapper[2553]: E0702 18:56:51.215451    2553 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f\": container with ID starting with 686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f not found: ID does not exist" containerID="686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"
Jul 02 18:56:51.215520 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:51.215475    2553 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f"} err="failed to get container status \"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f\": rpc error: code = NotFound desc = could not find container \"686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f\": container with ID starting with 686977c581ce462b5869b706c54e462da79edeecee286cc66cc031be295a069f not found: ID does not exist"
Jul 02 18:56:52.162128 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:52.162093    2553 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" path="/var/lib/kubelet/pods/0e4e065d-614d-4115-9ece-90f87b70cacb/volumes"
Jul 02 18:56:52.643169 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:52.643091    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-lhx8q_f0b5acde-5fb9-4fbc-8a51-589ec242aa45/cluster-monitoring-operator/0.log"
Jul 02 18:56:52.930333 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:52.930311    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/node-exporter/0.log"
Jul 02 18:56:52.957772 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:52.957749    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/kube-rbac-proxy/0.log"
Jul 02 18:56:52.985199 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:52.985174    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9rrwt_eb374ce5-64e9-4e57-b999-0617bd1b90a8/init-textfile/0.log"
Jul 02 18:56:55.060465 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:55.060438    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-b2ss9_1a3ae2c5-aebc-4d0b-873c-b9ef73bc8c20/networking-console-plugin/0.log"
Jul 02 18:56:55.593612 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:55.593581    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/2.log"
Jul 02 18:56:55.597513 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:55.597493    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-ctnnf_703b350d-5221-4434-b101-fe395ca946db/console-operator/3.log"
Jul 02 18:56:56.525949 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.525921    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-ftjdp_f0979127-7619-4133-a067-e6f9c0078edd/volume-data-source-validator/0.log"
Jul 02 18:56:56.857087 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857011    2553 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"]
Jul 02 18:56:56.857309 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857295    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="copy"
Jul 02 18:56:56.857350 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857312    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="copy"
Jul 02 18:56:56.857350 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857327    2553 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="gather"
Jul 02 18:56:56.857350 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857333    2553 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="gather"
Jul 02 18:56:56.857447 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857396    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="gather"
Jul 02 18:56:56.857447 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.857407    2553 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e4e065d-614d-4115-9ece-90f87b70cacb" containerName="copy"
Jul 02 18:56:56.859269 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.859254    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:56.861669 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.861649    2553 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m256x\"/\"default-dockercfg-6692h\""
Jul 02 18:56:56.861777 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.861759    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m256x\"/\"openshift-service-ca.crt\""
Jul 02 18:56:56.862829 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.862812    2553 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m256x\"/\"kube-root-ca.crt\""
Jul 02 18:56:56.867359 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.867338    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"]
Jul 02 18:56:56.945716 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.945676    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmvcc\" (UniqueName: \"kubernetes.io/projected/c4255278-2662-4fae-b18d-93c37c317a60-kube-api-access-pmvcc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:56.945908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.945732    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-lib-modules\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:56.945908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.945818    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-sys\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:56.945908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.945848    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-proc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:56.945908 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:56.945865    2553 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-podres\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047233 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047187    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmvcc\" (UniqueName: \"kubernetes.io/projected/c4255278-2662-4fae-b18d-93c37c317a60-kube-api-access-pmvcc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047233 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047249    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-lib-modules\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047280    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-sys\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047297    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-proc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047313    2553 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-podres\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047391    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-sys\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047435    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-proc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047455    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-podres\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.047486 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.047464    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c4255278-2662-4fae-b18d-93c37c317a60-lib-modules\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.055395 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.055357    2553 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmvcc\" (UniqueName: \"kubernetes.io/projected/c4255278-2662-4fae-b18d-93c37c317a60-kube-api-access-pmvcc\") pod \"perf-node-gather-daemonset-qp5js\" (UID: \"c4255278-2662-4fae-b18d-93c37c317a60\") " pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.169982 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.169948    2553 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:57.297367 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.297342    2553 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"]
Jul 02 18:56:57.300138 ip-10-0-130-27 kubenswrapper[2553]: W0702 18:56:57.300110    2553 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc4255278_2662_4fae_b18d_93c37c317a60.slice/crio-ac5612cd9c75d0bad95d4a0276194a9881422d59f0d8238d415a850e80a2fcce WatchSource:0}: Error finding container ac5612cd9c75d0bad95d4a0276194a9881422d59f0d8238d415a850e80a2fcce: Status 404 returned error can't find the container with id ac5612cd9c75d0bad95d4a0276194a9881422d59f0d8238d415a850e80a2fcce
Jul 02 18:56:57.472875 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.472788    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wbwr_93a17f0a-0ba7-45e1-ae42-f87237ee3e7e/dns/0.log"
Jul 02 18:56:57.493500 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.493466    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5wbwr_93a17f0a-0ba7-45e1-ae42-f87237ee3e7e/kube-rbac-proxy/0.log"
Jul 02 18:56:57.587469 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:57.587440    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-fxjbg_a3da5140-4064-4577-8191-4667846a5726/dns-node-resolver/0.log"
Jul 02 18:56:58.072432 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.072404    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27jxq_edc54bf1-8835-4198-91aa-4a09c593c2df/node-ca/0.log"
Jul 02 18:56:58.229280 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.229247    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js" event={"ID":"c4255278-2662-4fae-b18d-93c37c317a60","Type":"ContainerStarted","Data":"be3bc41e2a9e5a5ff10f27b165106ac15ed33bfefd4812313fb2e5152e8a8633"}
Jul 02 18:56:58.229436 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.229300    2553 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js" event={"ID":"c4255278-2662-4fae-b18d-93c37c317a60","Type":"ContainerStarted","Data":"ac5612cd9c75d0bad95d4a0276194a9881422d59f0d8238d415a850e80a2fcce"}
Jul 02 18:56:58.229436 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.229336    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:56:58.246894 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.246842    2553 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js" podStartSLOduration=2.246824699 podStartE2EDuration="2.246824699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 18:56:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 18:56:58.244415764 +0000 UTC m=+1594.573665387" watchObservedRunningTime="2026-07-02 18:56:58.246824699 +0000 UTC m=+1594.576074321"
Jul 02 18:56:58.905512 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:58.905477    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-674c7d54dc-s8b5k_97f2cfcc-f466-4c0f-a731-0adf39a75625/router/0.log"
Jul 02 18:56:59.286909 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.286864    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hjr72_70b55c16-655f-41b4-aea8-ea598a609543/serve-healthcheck-canary/0.log"
Jul 02 18:56:59.685413 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.685381    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-6h2w9_88ac6aaf-06fa-4909-ae0d-14b8849883b8/insights-operator/0.log"
Jul 02 18:56:59.686810 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.686790    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-6h2w9_88ac6aaf-06fa-4909-ae0d-14b8849883b8/insights-operator/1.log"
Jul 02 18:56:59.773386 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.773355    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zh2zr_9a844e35-052f-48ba-bb97-675804539124/kube-rbac-proxy/0.log"
Jul 02 18:56:59.793759 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.793731    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zh2zr_9a844e35-052f-48ba-bb97-675804539124/exporter/0.log"
Jul 02 18:56:59.813240 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:56:59.813217    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zh2zr_9a844e35-052f-48ba-bb97-675804539124/extractor/0.log"
Jul 02 18:57:02.054835 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.054807    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-npsnd_e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554/s3-init/0.log"
Jul 02 18:57:02.072725 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.072693    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-npsnd_e70dce61-d9ea-4a5e-9dc6-b48e1e1f4554/download-hf-model/0.log"
Jul 02 18:57:02.102234 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.102208    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-v68qm_93f315b1-e535-4fbe-b5c9-f414fde9f116/s3-tls-init-custom/0.log"
Jul 02 18:57:02.119100 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.119078    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-v68qm_93f315b1-e535-4fbe-b5c9-f414fde9f116/download-hf-model/0.log"
Jul 02 18:57:02.144549 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.144525    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-lnbvt_fa715faa-69a5-494d-b8fc-2f0b4226abf1/s3-tls-init-serving/0.log"
Jul 02 18:57:02.160214 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:02.160186    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-lnbvt_fa715faa-69a5-494d-b8fc-2f0b4226abf1/download-hf-model/0.log"
Jul 02 18:57:04.243955 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:04.243933    2553 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m256x/perf-node-gather-daemonset-qp5js"
Jul 02 18:57:06.014836 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:06.014743    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-fqslg_a1be429c-7b16-4375-ac8d-d5a97f8dfb54/migrator/0.log"
Jul 02 18:57:06.033571 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:06.033551    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-fqslg_a1be429c-7b16-4375-ac8d-d5a97f8dfb54/graceful-termination/0.log"
Jul 02 18:57:06.425726 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:06.425696    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-kh477_fe01e806-7bf3-4328-864c-e78edf501e2f/kube-storage-version-migrator-operator/1.log"
Jul 02 18:57:06.426604 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:06.426583    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-kh477_fe01e806-7bf3-4328-864c-e78edf501e2f/kube-storage-version-migrator-operator/0.log"
Jul 02 18:57:07.833559 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.833532    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/kube-multus-additional-cni-plugins/0.log"
Jul 02 18:57:07.854594 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.854572    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/egress-router-binary-copy/0.log"
Jul 02 18:57:07.879123 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.879103    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/cni-plugins/0.log"
Jul 02 18:57:07.901056 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.901035    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/bond-cni-plugin/0.log"
Jul 02 18:57:07.923443 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.923426    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/routeoverride-cni/0.log"
Jul 02 18:57:07.946250 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.946235    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/whereabouts-cni-bincopy/0.log"
Jul 02 18:57:07.968084 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:07.968062    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vmncx_9adbcc24-afe2-400e-8524-fad68fdc6797/whereabouts-cni/0.log"
Jul 02 18:57:08.048042 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:08.048018    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wqwvf_bd3cbe02-88e1-49b0-ae1e-063d5fceda7a/kube-multus/0.log"
Jul 02 18:57:08.066928 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:08.066908    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48vdd_86a22131-0126-4643-a945-41cbd09417dc/network-metrics-daemon/0.log"
Jul 02 18:57:08.092922 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:08.092902    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-48vdd_86a22131-0126-4643-a945-41cbd09417dc/kube-rbac-proxy/0.log"
Jul 02 18:57:09.281305 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.281266    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-controller/0.log"
Jul 02 18:57:09.297937 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.297904    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/0.log"
Jul 02 18:57:09.298088 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.298020    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovn-acl-logging/1.log"
Jul 02 18:57:09.313678 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.313660    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/kube-rbac-proxy-node/0.log"
Jul 02 18:57:09.333773 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.333754    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/kube-rbac-proxy-ovn-metrics/0.log"
Jul 02 18:57:09.353495 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.353475    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/northd/0.log"
Jul 02 18:57:09.372847 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.372819    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/nbdb/0.log"
Jul 02 18:57:09.391832 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.391799    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/sbdb/0.log"
Jul 02 18:57:09.435535 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:09.435502    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-n2nk2_2d6d1800-73c5-4a3f-951b-ab2b090cd3a7/ovnkube-controller/0.log"
Jul 02 18:57:10.560165 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:10.560139    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-989f8b8dc-drlq7_1986e221-4161-42ef-a43a-6fbef1f85c5e/check-endpoints/0.log"
Jul 02 18:57:10.610015 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:10.609991    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-kvrxf_61507973-7cb0-44d4-850c-05b89d4c27fc/network-check-target-container/0.log"
Jul 02 18:57:11.532349 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:11.532319    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-6cxpp_8eafaad5-84d2-4376-971b-e5e2b477e146/iptables-alerter/0.log"
Jul 02 18:57:12.158834 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:12.158809    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-wx2fv_3af6e73a-822a-4b59-8ad2-9bf6f5d61ad0/tuned/0.log"
Jul 02 18:57:13.816187 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:13.816158    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-b7q45_b21a829d-ec29-4eaf-9f02-5ab7fdb71643/cluster-samples-operator/0.log"
Jul 02 18:57:13.829682 ip-10-0-130-27 kubenswrapper[2553]: I0702 18:57:13.829662    2553 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-56fcf86bf-b7q45_b21a829d-ec29-4eaf-9f02-5ab7fdb71643/cluster-samples-operator-watch/0.log"