Jul 02 16:56:48.180157 ip-10-0-139-28 systemd[1]: Starting Kubernetes Kubelet...
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.642342    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653943    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653964    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653972    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653977    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653982    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653987    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653992    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:48.710416 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.653996    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654001    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654005    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654009    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654014    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654018    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654022    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654026    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654030    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654034    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654037    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654042    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654046    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654051    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654056    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654060    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654064    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654068    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654072    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654082    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:48.714366 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654086    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654091    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654094    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654098    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654102    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654106    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654111    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654115    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654118    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654122    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654127    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654131    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654135    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654139    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654143    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654147    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654151    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654155    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654158    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654162    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:48.717163 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654166    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654171    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654174    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654179    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654184    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654189    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654193    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654197    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654201    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654205    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654212    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654216    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654220    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654224    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654229    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654233    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654237    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654241    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654245    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654249    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:48.720773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654253    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654257    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654264    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654271    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654274    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654279    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654284    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654288    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654294    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654298    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654302    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654306    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654313    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654319    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654323    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654329    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654335    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654340    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654345    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:48.723667 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654349    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:48.722838 ip-10-0-139-28 systemd[1]: Started Kubernetes Kubelet.
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654353    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654356    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654361    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654365    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654369    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654374    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654378    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654382    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654386    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654390    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654394    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.654398    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656386    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656398    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656403    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656407    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656412    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656420    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:48.727735 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656427    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656431    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656435    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656439    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656443    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656447    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656451    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656455    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656458    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656462    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656467    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656473    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656478    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656482    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656486    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656490    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656494    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656499    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656504    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656508    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:48.730005 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656513    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656517    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656521    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656524    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656528    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656532    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656536    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656540    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656543    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656549    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656554    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656558    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656563    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656568    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656571    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656575    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656579    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656583    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656587    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:48.736446 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656591    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656595    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656599    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656603    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656606    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656610    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656614    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656618    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656622    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656625    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656630    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656634    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656639    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656643    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656647    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656650    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656656    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656659    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656663    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656667    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:48.739152 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656671    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656675    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656678    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656683    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656687    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656691    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656697    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656701    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656705    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656708    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656712    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656716    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656720    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656724    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656728    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656732    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656736    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656740    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656744    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656748    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:48.743414 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656752    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656756    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656760    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656764    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656768    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656773    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656777    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656781    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656785    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656789    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656796    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656799    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656804    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.656807    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657387    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657395    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657400    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657404    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657410    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657414    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:48.746811 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657419    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657422    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657426    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657430    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657434    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657438    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657445    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657450    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657454    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657459    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657464    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657468    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657472    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657476    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657480    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657484    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657488    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657493    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657496    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:48.753032 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657500    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657504    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657508    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657515    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657519    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657522    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657526    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657530    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657534    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657538    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657541    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657545    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657554    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657558    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657564    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657569    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657573    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657576    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657580    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657584    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:48.756239 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657588    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657591    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657595    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657604    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657608    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657614    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657619    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657623    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657627    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657632    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657636    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657640    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657645    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657649    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657653    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657659    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657663    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657667    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657671    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657675    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:48.760346 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657679    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657683    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657687    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657690    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657694    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657699    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657703    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657707    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657710    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657714    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657718    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657722    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657725    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657729    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657733    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657740    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657745    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657749    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657752    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657756    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:48.762410 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657760    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657764    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657767    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657771    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657775    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657779    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657783    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657788    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657792    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657796    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657800    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657804    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657808    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.657812    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657920    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657930    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657939    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657945    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657953    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657959    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657965    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 02 16:56:48.916252 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657972    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657976    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657981    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657987    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657992    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.657997    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658004    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658008    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658013    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658017    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658021    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658029    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658033    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658038    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658044    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658050    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658056    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658060    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658067    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658072    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658077    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658082    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658086    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658091    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 02 16:56:48.935259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658096    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658103    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658108    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658112    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658116    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658121    2581 flags.go:64] FLAG: --enable-server="true"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658126    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658133    2581 flags.go:64] FLAG: --event-burst="100"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658138    2581 flags.go:64] FLAG: --event-qps="50"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658142    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658146    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658151    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658156    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658161    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658167    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658171    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658175    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658180    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658184    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658189    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658193    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658197    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658202    2581 flags.go:64] FLAG: --feature-gates=""
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658208    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658212    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 02 16:56:48.937636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658217    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658223    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658228    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658232    2581 flags.go:64] FLAG: --help="false"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658236    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-139-28.ec2.internal"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658241    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658245    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658250    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658255    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658260    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658264    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658268    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658272    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658277    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658282    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658286    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658291    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658295    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658300    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658304    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658308    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658314    2581 flags.go:64] FLAG: --lock-file=""
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658318    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658323    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 02 16:56:48.940051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658327    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658336    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658340    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658344    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658348    2581 flags.go:64] FLAG: --logging-format="text"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658353    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658358    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658362    2581 flags.go:64] FLAG: --manifest-url=""
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658366    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658375    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658380    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658386    2581 flags.go:64] FLAG: --max-pods="110"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658390    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658394    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658399    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658404    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658408    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658412    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658417    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658428    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658433    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658437    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658441    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 02 16:56:48.942423 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658447    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8a2632cd31d78133be7b5fb6999469b2acf9b98d077c7f6ae5d0d94083d76b04"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658454    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658458    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658463    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658468    2581 flags.go:64] FLAG: --port="10250"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658472    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658477    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05efaa2fd1b0bd547"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658482    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658486    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658491    2581 flags.go:64] FLAG: --register-node="true"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658495    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658501    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658506    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658510    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658514    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658519    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658524    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658528    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658532    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658537    2581 flags.go:64] FLAG: --runonce="false"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658541    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658546    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658550    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658555    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658559    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658563    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 02 16:56:48.949320 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658568    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658573    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658577    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658582    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658585    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658590    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658594    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658600    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658604    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658612    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658616    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658620    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658629    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658633    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658637    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658641    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658646    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658650    2581 flags.go:64] FLAG: --v="2"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658656    2581 flags.go:64] FLAG: --version="false"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658662    2581 flags.go:64] FLAG: --vmodule=""
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658668    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.658673    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658845    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658851    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:48.950100 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658871    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658879    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658883    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658887    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658891    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658895    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658899    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658903    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658907    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658911    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658914    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658918    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658925    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658930    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658934    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658938    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658943    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658947    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658951    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:48.960228 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658955    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658960    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658964    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658967    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658971    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658975    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658979    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658982    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658984    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658987    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658990    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658992    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658995    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.658998    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659001    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659004    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659007    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659009    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659012    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659014    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:48.961008 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659018    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659022    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659027    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659031    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659035    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659039    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659043    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659047    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659051    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659055    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659060    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659064    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659067    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659072    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659076    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659079    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659083    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659087    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659091    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659095    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:48.961783 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659099    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659104    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659109    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659114    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659118    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659122    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659149    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659155    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659160    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659164    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659168    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659172    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659176    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659180    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659185    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659190    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659194    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659197    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659203    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:48.962563 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659208    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659212    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659216    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659219    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659224    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659229    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659235    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659238    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659242    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659246    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659250    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659254    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659259    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659263    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659266    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659271    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659275    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659278    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:48.963335 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.659282    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.660060    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.667431    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.667449    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.667472    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.667479    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667536    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667540    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667544    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667546    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667549    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667552    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667557    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667561    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:48.964535 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667565    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667569    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667572    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667574    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667577    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667579    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667582    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667585    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667588    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667591    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667596    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667599    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667602    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667605    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667608    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667610    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667613    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667616    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667619    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667622    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:48.965292 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667624    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667627    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667630    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667633    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667635    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667638    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667640    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667643    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667646    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667649    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667651    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667654    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667656    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667659    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667662    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667664    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667667    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667669    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667672    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667675    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:48.966072 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667678    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667680    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667686    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667688    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667691    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667694    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667697    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667699    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667702    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667705    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667708    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667711    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667713    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667716    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667719    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667722    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667724    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667727    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667729    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:48.966837 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667732    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667737    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667740    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667743    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667745    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667748    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667751    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667754    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667757    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667759    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667762    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667765    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667768    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667771    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667773    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667777    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667780    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667783    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667788    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:48.967615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667791    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667793    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667796    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667799    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667801    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667804    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667806    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667809    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667811    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667816    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667818    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667821    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667824    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.667829    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667968    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 02 16:56:48.968409 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667974    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667978    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667981    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667984    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667987    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.667996    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668000    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668003    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668006    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668009    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668011    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668014    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668017    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668020    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668024    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668026    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668030    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668033    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668035    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668038    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 02 16:56:49.066087 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668040    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668043    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668046    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668048    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668051    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668054    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668056    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668060    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668063    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668065    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668068    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668070    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668073    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668075    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668078    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668081    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668083    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668085    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668088    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668090    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 02 16:56:49.066831 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668093    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668096    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668098    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668101    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668103    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668106    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668110    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668113    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668116    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668119    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668121    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668124    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668127    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668131    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668138    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668142    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668144    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668148    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668151    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 02 16:56:49.067622 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668155    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668158    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668161    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668164    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668166    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668169    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668172    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668175    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668178    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668181    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668183    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668186    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668188    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668191    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668193    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668196    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668199    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668202    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668204    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668208    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 02 16:56:49.068429 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668211    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668214    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668217    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668220    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668222    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668225    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668227    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668230    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668232    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668235    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668238    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668240    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668243    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668245    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668248    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668251    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668254    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668258    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 02 16:56:49.069230 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:48.668261    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.668266    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.668380    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.670976    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.672627    2581 server.go:1019] "Starting client certificate rotation"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.672729    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.672768    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.675452    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.676593    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.687827    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.690377    2581 log.go:25] "Validated CRI v1 image API"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.691137    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 02 16:56:49.070056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.693659    2581 fs.go:135] Filesystem UUIDs: map[41e23fb9-a05e-46f4-b49d-544378830db5:/dev/nvme0n1p4 58b8edc8-e297-4311-9b54-82c431de89e1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 02 16:56:49.070444 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.693673    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.699542    2581 manager.go:216] Machine: {Timestamp:2026-07-02 16:56:48.698182172 +0000 UTC m=+0.385183155 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099490 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23adff3c22a49f6f499e6e20b40022 SystemUUID:ec23adff-3c22-a49f-6f49-9e6e20b40022 BootID:1370a392-7a15-4b54-a623-7b67de4ccbcb Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f9:cb:b2:21:b9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f9:cb:b2:21:b9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2e:0d:cc:91:03:2a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.699636    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.699713    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.124.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260623-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.703463    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.703495    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.704292    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.704303    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 02 16:56:49.070867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.704310    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.705179    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.706355    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.706491    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.710973    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711123    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711163    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711177    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711197    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711212    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711225    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711665    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.711685    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.713952    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.715836    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.715883    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716699    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716712    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716718    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716723    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716740    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716746    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716753    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716759    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716766    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716771    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716785    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.716790    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.717727    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.717742    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.720964    2581 server.go:1268] "Started kubelet"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.721201    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.721261    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.721051    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.722728    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.729182    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.730093    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.730200    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-28.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.730539    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 02 16:56:49.071260 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.731275    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.738304    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.738311    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.738393    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739054    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739059    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739076    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739056    2581 factory.go:55] Registering systemd factory
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739220    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739230    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739427    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739764    2581 factory.go:153] Registering CRI-O factory
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739776    2581 factory.go:223] Registration of the crio container factory successfully
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739819    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739837    2581 factory.go:103] Registering Raw factory
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.739847    2581 manager.go:1196] Started watching for new ooms in manager
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.740214    2581 manager.go:349] Starting recovery of all containers
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.739371    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-28.ec2.internal.18be876def47cc93  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-28.ec2.internal,UID:ip-10-0-139-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-28.ec2.internal,},FirstTimestamp:2026-07-02 16:56:48.720940179 +0000 UTC m=+0.407941141,LastTimestamp:2026-07-02 16:56:48.720940179 +0000 UTC m=+0.407941141,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-28.ec2.internal,}"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.742948    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.743049    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.746682    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.749750    2581 manager.go:354] Recovery completed
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.756971    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2qsdv"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.759639    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.761448    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.761479    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.761493    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762005    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762016    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762031    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762046    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762773    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2qsdv"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762812    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.075708 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762835    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.762849    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.763550    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-28.ec2.internal.18be876df1b21bc7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-28.ec2.internal,UID:ip-10-0-139-28.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-28.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-28.ec2.internal,},FirstTimestamp:2026-07-02 16:56:48.761461703 +0000 UTC m=+0.448462672,LastTimestamp:2026-07-02 16:56:48.761461703 +0000 UTC m=+0.448462672,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-28.ec2.internal,}"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.764171    2581 policy_none.go:49] "None policy: Start"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.764188    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.764204    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.766202    2581 policy_none.go:47] "Start"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.790420    2581 manager.go:337] "Starting Device Plugin manager"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.790452    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.790466    2581 server.go:87] "Starting device plugin registration server"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.790812    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.790823    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.790958    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.791048    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.791054    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.791624    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.791678    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.809417    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.809454    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.809478    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.809513    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.820408    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.891174    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.892252    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.892279    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.892288    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.892335    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.899193    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.899210    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-28.ec2.internal\": node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.909948    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"]
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.910001    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.911547    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.076926 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.911577    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.911596    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.913878    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.913979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914004    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914641    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914662    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914674    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914733    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914754    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.914763    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.916909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.916940    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.917619    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientMemory"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.917643    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasNoDiskPressure"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:48.917655    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeHasSufficientPID"
Jul 02 16:56:49.078928 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:48.927413    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:49.831980    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832051    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832023    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 16:51:49 +0000 UTC" deadline="2028-02-23 21:39:25.186733093 +0000 UTC"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832094    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14428h42m35.354643768s"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e757ec6aacc35d99d6a9719bfafce01a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-28.ec2.internal\" (UID: \"e757ec6aacc35d99d6a9719bfafce01a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832172    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.832293    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection force closed via ClientConn.Close\") has prevented the request from succeeding"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:49.832385    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://a0982126e3c4d4300aa49bdc444c76df-0aa7d40c4aa2717d.elb.us-east-1.amazonaws.com:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/ip-10-0-139-28.ec2.internal?timeout=10s\": http2: client connection force closed via ClientConn.Close" interval="400ms"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:49.860972    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-28.ec2.internal\" not found" node="ip-10-0-139-28.ec2.internal"
Jul 02 16:56:49.908954 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:49.865748    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-28.ec2.internal\" not found" node="ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:49.932898    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-28.ec2.internal\" not found"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.932968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e757ec6aacc35d99d6a9719bfafce01a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-28.ec2.internal\" (UID: \"e757ec6aacc35d99d6a9719bfafce01a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.932995    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.933016    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.933070    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.933072    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e757ec6aacc35d99d6a9719bfafce01a-config\") pod \"kube-apiserver-proxy-ip-10-0-139-28.ec2.internal\" (UID: \"e757ec6aacc35d99d6a9719bfafce01a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.933100    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bd323a7b0d15c1fb472752ef1a5a873f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal\" (UID: \"bd323a7b0d15c1fb472752ef1a5a873f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.964235    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 16:56:50.038360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:49.992198    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 16:56:50.039672 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.039173    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.182537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.047924    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 16:56:50.182537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.055546    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.182537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.068380    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 02 16:56:50.182537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.168542    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.182537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.170817    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal"
Jul 02 16:56:50.318812 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.318721    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 16:56:50.832474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.832432    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 02 16:56:50.832474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.832457    2581 apiserver.go:52] "Watching apiserver"
Jul 02 16:56:50.842140 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.842111    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 02 16:56:50.842513 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.842489    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp","openshift-cluster-node-tuning-operator/tuned-572tv","openshift-multus/network-metrics-daemon-srp4c","openshift-network-diagnostics/network-check-target-565sb","kube-system/konnectivity-agent-wnmxk","openshift-dns/node-resolver-k8bnn","openshift-image-registry/node-ca-g5t5n","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal","openshift-multus/multus-additional-cni-plugins-9s2wn","openshift-multus/multus-s77nv","openshift-network-operator/iptables-alerter-ct7pg","openshift-ovn-kubernetes/ovnkube-node-x9mwx","kube-system/global-pull-secret-syncer-hx8nl"]
Jul 02 16:56:50.844967 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.844948    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:50.846419 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.846401    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 02 16:56:50.847283 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.847251    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.847904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.847878    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 02 16:56:50.847904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.847878    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 02 16:56:50.848041 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.848007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rvxnr\""
Jul 02 16:56:50.849520 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.849505    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.849961 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.849943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 02 16:56:50.849961 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.849954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wk22h\""
Jul 02 16:56:50.850091 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.850079    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.850212 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.850196    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.851950 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.851914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.852054 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.851964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.852124 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.852032    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.852181 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.852050    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fz57q\""
Jul 02 16:56:50.854505 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.854490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.854760 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.854742    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.856000 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.855520    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jd5tb\""
Jul 02 16:56:50.856000 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.855743    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 02 16:56:50.856151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.856114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 02 16:56:50.856151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.856127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:50.858406 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.858389    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:50.858792 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:50.858689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:56:50.859285 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.859262    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nsfc2\""
Jul 02 16:56:50.859388 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.859370    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.859660 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.859641    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.861038 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.861023    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:50.863420 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.863403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.863650 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.863633    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.863971 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.863957    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-jf8k9\""
Jul 02 16:56:50.864056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.863970    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.864056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.863973    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 02 16:56:50.865863 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.865847    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:50.865962 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:50.865938    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:56:50.866107 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.866089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 02 16:56:50.866355 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.866339    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-5nnvx\""
Jul 02 16:56:50.866422 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.866357    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 02 16:56:50.868311 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.868286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:50.871068 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.871045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.871162 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.871148    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.871283 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.871261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 02 16:56:50.871414 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.871394    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-59qcx\""
Jul 02 16:56:50.871925 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.871674    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.873687 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.873666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:50.873787 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:50.873717    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:56:50.874135 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.874114    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 02 16:56:50.874346 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.874329    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 02 16:56:50.874427 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.874387    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 02 16:56:50.874518 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.874505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-snqsg\""
Jul 02 16:56:50.875588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.875571    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 02 16:56:50.875670 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.875614    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 02 16:56:50.875670 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.875657    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 02 16:56:50.883003 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.882982    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jj5fc"
Jul 02 16:56:50.889453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.889434    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jj5fc"
Jul 02 16:56:50.892004 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:50.891966    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd323a7b0d15c1fb472752ef1a5a873f.slice/crio-52ebaef64abcd970cdd709aecfef8ca27dac7873b8cb150b3cbcdb9717c6a9c7 WatchSource:0}: Error finding container 52ebaef64abcd970cdd709aecfef8ca27dac7873b8cb150b3cbcdb9717c6a9c7: Status 404 returned error can't find the container with id 52ebaef64abcd970cdd709aecfef8ca27dac7873b8cb150b3cbcdb9717c6a9c7
Jul 02 16:56:50.898318 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.898302    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 16:56:50.919432 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:50.919279    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode757ec6aacc35d99d6a9719bfafce01a.slice/crio-111c0c2574384cd28ac700803c8c63fba7d76df6df69214e76999dc746f04eca WatchSource:0}: Error finding container 111c0c2574384cd28ac700803c8c63fba7d76df6df69214e76999dc746f04eca: Status 404 returned error can't find the container with id 111c0c2574384cd28ac700803c8c63fba7d76df6df69214e76999dc746f04eca
Jul 02 16:56:50.940479 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.940457    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 02 16:56:50.947362 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-run\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.947484 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947373    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-kubelet\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.947484 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947400    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.947484 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-iptables-alerter-script\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:50.947484 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-kubelet\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.947484 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947476    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947501    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-bin\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947528    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25be8fe2-195c-4ff1-ab86-cefa434f6eae-agent-certs\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-sys\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-hostroot\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947607    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysconfig\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq4q5\" (UniqueName: \"kubernetes.io/projected/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-kube-api-access-sq4q5\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947655    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-netns\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.947732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947681    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg959\" (UniqueName: \"kubernetes.io/projected/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-kube-api-access-cg959\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947728    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-systemd-units\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947766    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-node-log\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947784    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-netd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-lib-modules\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947849    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-kubelet-config\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947941    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-device-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947961    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-systemd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.947992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-sys-fs\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948016    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-system-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.948042 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-daemon-config\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948071    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-os-release\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948096    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948113    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-host-slash\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-var-lib-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948142    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f23d5a2-8e77-4895-ab64-361cd53fbc90-hosts-file\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-multus\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948208    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-netns\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948223    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-script-lib\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wd52\" (UniqueName: \"kubernetes.io/projected/51bec5ed-8842-47d8-8faa-9be7933368a9-kube-api-access-4wd52\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948263    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25be8fe2-195c-4ff1-ab86-cefa434f6eae-konnectivity-ca\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:50.948474 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948303    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-modprobe-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-etc-tuned\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d5a2-8e77-4895-ab64-361cd53fbc90-tmp-dir\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948399    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-tmp\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948416    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-etc-selinux\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-conf-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmcj8\" (UniqueName: \"kubernetes.io/projected/855cb017-d9f0-47da-800c-b325b3067bad-kube-api-access-hmcj8\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948489    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-log-socket\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948508    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-env-overrides\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948536    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-kubernetes\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948566    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-conf\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948586    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-registration-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948604    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6t8g\" (UniqueName: \"kubernetes.io/projected/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kube-api-access-s6t8g\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948628    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cni-binary-copy\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.949037 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f6kx\" (UniqueName: \"kubernetes.io/projected/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-kube-api-access-5f6kx\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948666    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855cb017-d9f0-47da-800c-b325b3067bad-host\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948682    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-cnibin\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948696    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-var-lib-kubelet\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948711    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948726    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-system-cni-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-etc-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948778    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-config\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948813    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51bec5ed-8842-47d8-8faa-9be7933368a9-ovn-node-metrics-cert\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-host\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948894    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cnibin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-bin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.949588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-systemd\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.948997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-socket-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949023    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7vvr\" (UniqueName: \"kubernetes.io/projected/9b0f716e-c53d-401c-8f26-e65f68a94306-kube-api-access-m7vvr\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949068    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/855cb017-d9f0-47da-800c-b325b3067bad-serviceca\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-dbus\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-os-release\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949163    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-socket-dir-parent\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-multus-certs\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949211    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpt6\" (UniqueName: \"kubernetes.io/projected/9f23d5a2-8e77-4895-ab64-361cd53fbc90-kube-api-access-qhpt6\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-k8s-cni-cncf-io\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-etc-kubernetes\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:50.950083 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82d5f\" (UniqueName: \"kubernetes.io/projected/db355805-3006-4ea1-aee2-8317c712c3b5-kube-api-access-82d5f\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:50.950498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-slash\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:50.950498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:50.949386    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-ovn\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.050498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050412    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-modprobe-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-etc-tuned\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d5a2-8e77-4895-ab64-361cd53fbc90-tmp-dir\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.050498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050485    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-tmp\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050498 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-etc-selinux\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-conf-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050543    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmcj8\" (UniqueName: \"kubernetes.io/projected/855cb017-d9f0-47da-800c-b325b3067bad-kube-api-access-hmcj8\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050601    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-log-socket\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050630    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-modprobe-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-env-overrides\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-kubernetes\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050701    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-conf-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050707    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-conf\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050735    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-registration-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050763    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6t8g\" (UniqueName: \"kubernetes.io/projected/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kube-api-access-s6t8g\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050791    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cni-binary-copy\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050846    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f6kx\" (UniqueName: \"kubernetes.io/projected/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-kube-api-access-5f6kx\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855cb017-d9f0-47da-800c-b325b3067bad-host\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.050876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050867    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f23d5a2-8e77-4895-ab64-361cd53fbc90-tmp-dir\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050905    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-log-socket\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-cnibin\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-var-lib-kubelet\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050998    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/855cb017-d9f0-47da-800c-b325b3067bad-host\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051022    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-var-lib-kubelet\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.050860    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-etc-selinux\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051087    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-cnibin\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051250    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-env-overrides\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051265    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-kubernetes\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051273    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051306    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-registration-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051425    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-conf\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.051601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-system-cni-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.051463    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-etc-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051492    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-system-cni-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-config\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051539    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-etc-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.051545    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:51.551496545 +0000 UTC m=+3.238497496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051571    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51bec5ed-8842-47d8-8faa-9be7933368a9-ovn-node-metrics-cert\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051566    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cni-binary-copy\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-host\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cnibin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051755    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-bin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051784    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.052374 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051811    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-systemd\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051837    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-bin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051847    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-cnibin\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-socket-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051914    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-host\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7vvr\" (UniqueName: \"kubernetes.io/projected/9b0f716e-c53d-401c-8f26-e65f68a94306-kube-api-access-m7vvr\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051942    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051967    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/855cb017-d9f0-47da-800c-b325b3067bad-serviceca\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-socket-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.051991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052013    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-dbus\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052034    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-os-release\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052037    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-systemd\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-socket-dir-parent\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-multus-certs\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052103    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpt6\" (UniqueName: \"kubernetes.io/projected/9f23d5a2-8e77-4895-ab64-361cd53fbc90-kube-api-access-qhpt6\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-k8s-cni-cncf-io\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052173    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-etc-kubernetes\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052194    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82d5f\" (UniqueName: \"kubernetes.io/projected/db355805-3006-4ea1-aee2-8317c712c3b5-kube-api-access-82d5f\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052218    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-slash\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052238    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-ovn\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-run\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052278    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-kubelet\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052326    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-kubelet\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysctl-d\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052457    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052461    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-iptables-alerter-script\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.053834 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-kubelet\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-k8s-cni-cncf-io\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052558    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-bin\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052580    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-etc-kubernetes\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052609    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-bin\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25be8fe2-195c-4ff1-ab86-cefa434f6eae-agent-certs\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/855cb017-d9f0-47da-800c-b325b3067bad-serviceca\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052650    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-sys\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052663    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052672    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-hostroot\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysconfig\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052695    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-kubelet\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052739    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-slash\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.054551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052742    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-ovn\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052766    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq4q5\" (UniqueName: \"kubernetes.io/projected/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-kube-api-access-sq4q5\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052772    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-run\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-netns\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052802    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-socket-dir-parent\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052815    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-hostroot\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-etc-sysconfig\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053085    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-iptables-alerter-script\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.052055    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-config\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053145    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-netns\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053144    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-dbus\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053173    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-run-multus-certs\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053220    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-os-release\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053228    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-sys\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg959\" (UniqueName: \"kubernetes.io/projected/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-kube-api-access-cg959\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.055277 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-systemd-units\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053306    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-binary-copy\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053324    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-node-log\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053346    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-netd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-node-log\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053392    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053417    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-systemd-units\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-lib-modules\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-kubelet-config\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053480    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-cni-netd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053488    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053497    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053521    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-device-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.055904 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053546    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a94f166e-7d6e-4716-9986-ea647fcb439c-kubelet-config\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-systemd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-sys-fs\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053589    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/db355805-3006-4ea1-aee2-8317c712c3b5-lib-modules\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-system-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.053642    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053647    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-daemon-config\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053657    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-run-systemd\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053668    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-device-dir\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.053707    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:56:51.55368291 +0000 UTC m=+3.240683868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053724    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-system-cni-dir\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053750    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-os-release\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053786    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/64d32f64-0814-4ab7-9d2a-ed25c1fee465-sys-fs\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.056453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053793    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-host-slash\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-var-lib-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053870    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053886    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9b0f716e-c53d-401c-8f26-e65f68a94306-os-release\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053932    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-host-slash\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.053941    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-ovn-kubernetes\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054004    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-var-lib-openvswitch\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f23d5a2-8e77-4895-ab64-361cd53fbc90-hosts-file\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-multus\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054177    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9f23d5a2-8e77-4895-ab64-361cd53fbc90-hosts-file\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-netns\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054235    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-script-lib\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-host-var-lib-cni-multus\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.056877 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wd52\" (UniqueName: \"kubernetes.io/projected/51bec5ed-8842-47d8-8faa-9be7933368a9-kube-api-access-4wd52\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054319    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/51bec5ed-8842-47d8-8faa-9be7933368a9-host-run-netns\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054329    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-multus-daemon-config\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054443    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25be8fe2-195c-4ff1-ab86-cefa434f6eae-konnectivity-ca\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.054487    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9b0f716e-c53d-401c-8f26-e65f68a94306-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.055030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/51bec5ed-8842-47d8-8faa-9be7933368a9-ovnkube-script-lib\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.055283    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/25be8fe2-195c-4ff1-ab86-cefa434f6eae-konnectivity-ca\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.056054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-tmp\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.056153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/db355805-3006-4ea1-aee2-8317c712c3b5-etc-tuned\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.056198    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/51bec5ed-8842-47d8-8faa-9be7933368a9-ovn-node-metrics-cert\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.057307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.056290    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/25be8fe2-195c-4ff1-ab86-cefa434f6eae-agent-certs\") pod \"konnectivity-agent-wnmxk\" (UID: \"25be8fe2-195c-4ff1-ab86-cefa434f6eae\") " pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:51.062425 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.062404    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmcj8\" (UniqueName: \"kubernetes.io/projected/855cb017-d9f0-47da-800c-b325b3067bad-kube-api-access-hmcj8\") pod \"node-ca-g5t5n\" (UID: \"855cb017-d9f0-47da-800c-b325b3067bad\") " pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.065878 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.065852    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f6kx\" (UniqueName: \"kubernetes.io/projected/5060ab2f-b2ce-4b34-8e08-2086e9af8e3c-kube-api-access-5f6kx\") pod \"iptables-alerter-ct7pg\" (UID: \"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c\") " pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.066203 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.066187    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6t8g\" (UniqueName: \"kubernetes.io/projected/64d32f64-0814-4ab7-9d2a-ed25c1fee465-kube-api-access-s6t8g\") pod \"aws-ebs-csi-driver-node-vv8gp\" (UID: \"64d32f64-0814-4ab7-9d2a-ed25c1fee465\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.069564 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.069528    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:56:51.069564 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.069556    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:56:51.069775 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.069571    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:51.069775 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.069639    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:51.569619127 +0000 UTC m=+3.256620078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:51.070158 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.070132    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7vvr\" (UniqueName: \"kubernetes.io/projected/9b0f716e-c53d-401c-8f26-e65f68a94306-kube-api-access-m7vvr\") pod \"multus-additional-cni-plugins-9s2wn\" (UID: \"9b0f716e-c53d-401c-8f26-e65f68a94306\") " pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.070773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.070183    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82d5f\" (UniqueName: \"kubernetes.io/projected/db355805-3006-4ea1-aee2-8317c712c3b5-kube-api-access-82d5f\") pod \"tuned-572tv\" (UID: \"db355805-3006-4ea1-aee2-8317c712c3b5\") " pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.070773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.070368    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpt6\" (UniqueName: \"kubernetes.io/projected/9f23d5a2-8e77-4895-ab64-361cd53fbc90-kube-api-access-qhpt6\") pod \"node-resolver-k8bnn\" (UID: \"9f23d5a2-8e77-4895-ab64-361cd53fbc90\") " pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.070773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.070548    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg959\" (UniqueName: \"kubernetes.io/projected/c42e8c85-0da3-4dea-8cca-51ee7e3b781d-kube-api-access-cg959\") pod \"multus-s77nv\" (UID: \"c42e8c85-0da3-4dea-8cca-51ee7e3b781d\") " pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.070773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.070686    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wd52\" (UniqueName: \"kubernetes.io/projected/51bec5ed-8842-47d8-8faa-9be7933368a9-kube-api-access-4wd52\") pod \"ovnkube-node-x9mwx\" (UID: \"51bec5ed-8842-47d8-8faa-9be7933368a9\") " pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.071773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.071753    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq4q5\" (UniqueName: \"kubernetes.io/projected/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-kube-api-access-sq4q5\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:51.115902 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.115856    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 02 16:56:51.160875 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.160845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:56:51.171615 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.171588    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25be8fe2_195c_4ff1_ab86_cefa434f6eae.slice/crio-472abe1a294c7a36766b24a53e0cd0b953082eb43c72572724fd162e15b85f18 WatchSource:0}: Error finding container 472abe1a294c7a36766b24a53e0cd0b953082eb43c72572724fd162e15b85f18: Status 404 returned error can't find the container with id 472abe1a294c7a36766b24a53e0cd0b953082eb43c72572724fd162e15b85f18
Jul 02 16:56:51.174990 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.174969    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp"
Jul 02 16:56:51.183594 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.183565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d32f64_0814_4ab7_9d2a_ed25c1fee465.slice/crio-54ab17e8ed597aeddd58c5e0f8467061005bb99c5887276922d73ae32974d21b WatchSource:0}: Error finding container 54ab17e8ed597aeddd58c5e0f8467061005bb99c5887276922d73ae32974d21b: Status 404 returned error can't find the container with id 54ab17e8ed597aeddd58c5e0f8467061005bb99c5887276922d73ae32974d21b
Jul 02 16:56:51.204869 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.204830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ct7pg"
Jul 02 16:56:51.209063 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.209040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-572tv"
Jul 02 16:56:51.215421 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.215394    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5060ab2f_b2ce_4b34_8e08_2086e9af8e3c.slice/crio-b516a4d25240e7329e07e557563c47157af3b3dd3d6db7f037449b5c5d65438c WatchSource:0}: Error finding container b516a4d25240e7329e07e557563c47157af3b3dd3d6db7f037449b5c5d65438c: Status 404 returned error can't find the container with id b516a4d25240e7329e07e557563c47157af3b3dd3d6db7f037449b5c5d65438c
Jul 02 16:56:51.215536 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.215485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-s77nv"
Jul 02 16:56:51.220312 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.220292    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb355805_3006_4ea1_aee2_8317c712c3b5.slice/crio-01d1aeb047e036ea634a9b1c8fa9fefc6afb9a47cf0cdafc9680faf3af4b95ce WatchSource:0}: Error finding container 01d1aeb047e036ea634a9b1c8fa9fefc6afb9a47cf0cdafc9680faf3af4b95ce: Status 404 returned error can't find the container with id 01d1aeb047e036ea634a9b1c8fa9fefc6afb9a47cf0cdafc9680faf3af4b95ce
Jul 02 16:56:51.222359 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.222338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-k8bnn"
Jul 02 16:56:51.224691 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.224668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc42e8c85_0da3_4dea_8cca_51ee7e3b781d.slice/crio-b15756057ff61a4e9f5a09efe936e6b2c338be1b4a226d63cae51ac62824e9d1 WatchSource:0}: Error finding container b15756057ff61a4e9f5a09efe936e6b2c338be1b4a226d63cae51ac62824e9d1: Status 404 returned error can't find the container with id b15756057ff61a4e9f5a09efe936e6b2c338be1b4a226d63cae51ac62824e9d1
Jul 02 16:56:51.227957 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.227917    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g5t5n"
Jul 02 16:56:51.230901 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.230881    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f23d5a2_8e77_4895_ab64_361cd53fbc90.slice/crio-d67c630b67beba7f67822d42436c969206d4d97f55077413974c0ed100d6127d WatchSource:0}: Error finding container d67c630b67beba7f67822d42436c969206d4d97f55077413974c0ed100d6127d: Status 404 returned error can't find the container with id d67c630b67beba7f67822d42436c969206d4d97f55077413974c0ed100d6127d
Jul 02 16:56:51.234804 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.234781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9s2wn"
Jul 02 16:56:51.235780 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.235760    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:56:51.236344 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.236324    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855cb017_d9f0_47da_800c_b325b3067bad.slice/crio-1059af4a97323802baa215230fd90dd66fe69f6eb1efb6fc6d849fa15c4571e1 WatchSource:0}: Error finding container 1059af4a97323802baa215230fd90dd66fe69f6eb1efb6fc6d849fa15c4571e1: Status 404 returned error can't find the container with id 1059af4a97323802baa215230fd90dd66fe69f6eb1efb6fc6d849fa15c4571e1
Jul 02 16:56:51.244661 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.244643    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0f716e_c53d_401c_8f26_e65f68a94306.slice/crio-d350080aa9a0beb3aa1c0da70b57977a4f029e5fc018431416d7d1580d7cbbe5 WatchSource:0}: Error finding container d350080aa9a0beb3aa1c0da70b57977a4f029e5fc018431416d7d1580d7cbbe5: Status 404 returned error can't find the container with id d350080aa9a0beb3aa1c0da70b57977a4f029e5fc018431416d7d1580d7cbbe5
Jul 02 16:56:51.249080 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:56:51.249058    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51bec5ed_8842_47d8_8faa_9be7933368a9.slice/crio-1932f448f3049b28c4750b42cfaedc5433228457afe081473f3fe249e3388a79 WatchSource:0}: Error finding container 1932f448f3049b28c4750b42cfaedc5433228457afe081473f3fe249e3388a79: Status 404 returned error can't find the container with id 1932f448f3049b28c4750b42cfaedc5433228457afe081473f3fe249e3388a79
Jul 02 16:56:51.557219 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.557167    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:51.557383 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.557258    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:51.557383 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.557296    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:51.557383 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.557339    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:51.557383 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.557366    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:56:52.557348904 +0000 UTC m=+4.244349858 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:51.557383 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.557384    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:52.557376851 +0000 UTC m=+4.244377802 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:51.658470 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.658424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:51.658665 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.658646    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:56:51.658753 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.658668    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:56:51.658753 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.658681    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:51.658753 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:51.658743    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:52.658723675 +0000 UTC m=+4.345724642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:51.840578 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.840471    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnmxk" event={"ID":"25be8fe2-195c-4ff1-ab86-cefa434f6eae","Type":"ContainerStarted","Data":"472abe1a294c7a36766b24a53e0cd0b953082eb43c72572724fd162e15b85f18"}
Jul 02 16:56:51.850726 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.850435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal" event={"ID":"e757ec6aacc35d99d6a9719bfafce01a","Type":"ContainerStarted","Data":"111c0c2574384cd28ac700803c8c63fba7d76df6df69214e76999dc746f04eca"}
Jul 02 16:56:51.857247 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.856943    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal" event={"ID":"bd323a7b0d15c1fb472752ef1a5a873f","Type":"ContainerStarted","Data":"52ebaef64abcd970cdd709aecfef8ca27dac7873b8cb150b3cbcdb9717c6a9c7"}
Jul 02 16:56:51.869508 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.869467    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerStarted","Data":"d350080aa9a0beb3aa1c0da70b57977a4f029e5fc018431416d7d1580d7cbbe5"}
Jul 02 16:56:51.891782 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.891741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g5t5n" event={"ID":"855cb017-d9f0-47da-800c-b325b3067bad","Type":"ContainerStarted","Data":"1059af4a97323802baa215230fd90dd66fe69f6eb1efb6fc6d849fa15c4571e1"}
Jul 02 16:56:51.892787 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.892717    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 16:51:51 +0000 UTC" deadline="2028-03-17 16:23:52.699583914 +0000 UTC"
Jul 02 16:56:51.892787 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.892779    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14975h27m0.806809167s"
Jul 02 16:56:51.918586 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.918489    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-572tv" event={"ID":"db355805-3006-4ea1-aee2-8317c712c3b5","Type":"ContainerStarted","Data":"01d1aeb047e036ea634a9b1c8fa9fefc6afb9a47cf0cdafc9680faf3af4b95ce"}
Jul 02 16:56:51.929760 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.929716    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ct7pg" event={"ID":"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c","Type":"ContainerStarted","Data":"b516a4d25240e7329e07e557563c47157af3b3dd3d6db7f037449b5c5d65438c"}
Jul 02 16:56:51.959601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.959559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"1932f448f3049b28c4750b42cfaedc5433228457afe081473f3fe249e3388a79"}
Jul 02 16:56:51.981403 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:51.981367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8bnn" event={"ID":"9f23d5a2-8e77-4895-ab64-361cd53fbc90","Type":"ContainerStarted","Data":"d67c630b67beba7f67822d42436c969206d4d97f55077413974c0ed100d6127d"}
Jul 02 16:56:52.008068 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.008024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s77nv" event={"ID":"c42e8c85-0da3-4dea-8cca-51ee7e3b781d","Type":"ContainerStarted","Data":"b15756057ff61a4e9f5a09efe936e6b2c338be1b4a226d63cae51ac62824e9d1"}
Jul 02 16:56:52.029023 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.028975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp" event={"ID":"64d32f64-0814-4ab7-9d2a-ed25c1fee465","Type":"ContainerStarted","Data":"54ab17e8ed597aeddd58c5e0f8467061005bb99c5887276922d73ae32974d21b"}
Jul 02 16:56:52.566339 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.566249    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:52.566339 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.566334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:52.566624 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.566456    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:52.566624 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.566515    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:54.566496495 +0000 UTC m=+6.253497449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:52.567087 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.566940    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:52.567087 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.567044    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:56:54.567026523 +0000 UTC m=+6.254027478 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:52.667516 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.667307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:52.667719 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.667543    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:56:52.667719 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.667564    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:56:52.667719 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.667576    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:52.667719 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.667657    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:54.667638122 +0000 UTC m=+6.354639091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:52.813825 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.813081    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:52.813825 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.813239    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:56:52.813825 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.813675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:52.813825 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.813781    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:56:52.814515 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.814316    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:52.814515 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:52.814413    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:56:52.893631 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.893533    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 16:51:51 +0000 UTC" deadline="2028-04-09 14:06:48.838510048 +0000 UTC"
Jul 02 16:56:52.893631 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:52.893580    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15525h9m55.944935206s"
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.585975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.586071    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.586232    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.586300    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:56:58.586279393 +0000 UTC m=+10.273280350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.586741    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:54.586839 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.586792    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:56:58.586777909 +0000 UTC m=+10.273778866 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:54.686455 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.686414    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:54.686685 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.686628    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:56:54.686685 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.686654    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:56:54.686685 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.686667    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:54.686882 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.686737    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:56:58.686716161 +0000 UTC m=+10.373717138 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:54.810032 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.810001    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:54.810233 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.810141    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:56:54.810565 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.810539    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:54.810685 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.810659    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:56:54.810748 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:54.810721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:54.810831 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:54.810805    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:56:56.810659 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:56.810625    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:56.810659 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:56.810641    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:56.811202 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:56.810776    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:56:56.811202 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:56.811156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:56.811308 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:56.811256    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:56:56.811369 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:56.811334    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:56:58.622165 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.622120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:58.622596 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.622224    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:58.623275 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.623248    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:58.623371 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.623349    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:06.623325398 +0000 UTC m=+18.310326368 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:56:58.626265 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.626232    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:58.626404 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.626320    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:06.626300763 +0000 UTC m=+18.313301717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:56:58.723497 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.723431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:58.723687 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.723571    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:56:58.723687 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.723594    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:56:58.723687 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.723609    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:58.723687 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.723671    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:06.723652641 +0000 UTC m=+18.410653603 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:56:58.812020 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.811983    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:56:58.812206 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.812161    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:56:58.812279 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:56:58.812217    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:56:58.812279 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.812244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:56:58.812394 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.812304    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:56:58.812394 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:56:58.812383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:00.810856 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:00.810323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:00.810856 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:00.810468    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:00.810856 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:00.810756    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:00.811660 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:00.810887    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:00.811660 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:00.811025    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:00.811660 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:00.811119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:02.810764 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:02.810731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:02.810764 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:02.810751    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:02.811190 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:02.810736    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:02.811190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:02.810879    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:02.811190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:02.810984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:02.811190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:02.811056    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:04.810682 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:04.810650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:04.811177 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:04.810650    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:04.811177 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:04.810784    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:04.811177 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:04.810811    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:04.811177 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:04.810907    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:04.811177 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:04.810977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:06.684539 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.684496    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:06.685051 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.684579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:06.685051 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.684609    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:57:06.685051 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.684687    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:22.684668677 +0000 UTC m=+34.371669633 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:57:06.685051 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.684687    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:57:06.685051 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.684741    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:22.684724889 +0000 UTC m=+34.371725856 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:57:06.785569 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.785535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:06.785762 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.785691    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:57:06.785762 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.785710    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:57:06.785762 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.785723    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:57:06.785934 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.785782    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:22.785764663 +0000 UTC m=+34.472765629 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:57:06.810585 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.810556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:06.810732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.810556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:06.810732 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.810687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:06.810853 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:06.810556    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:06.810853 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.810765    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:06.810968 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:06.810912    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:08.810680 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:08.810649    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:08.811149 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:08.810748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:08.811149 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:08.810782    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:08.811149 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:08.810827    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:08.811149 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:08.810876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:08.811149 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:08.810931    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:10.810231 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:10.810191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:10.810231 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:10.810198    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:10.810747 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:10.810397    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:10.810747 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:10.810448    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:10.810747 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:10.810604    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:10.810747 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:10.810700    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:12.810561 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:12.810340    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:12.810994 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:12.810341    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:12.810994 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:12.810679    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:12.810994 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:12.810380    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:12.810994 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:12.810758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:12.810994 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:12.810844    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:13.077072 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.077029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-572tv" event={"ID":"db355805-3006-4ea1-aee2-8317c712c3b5","Type":"ContainerStarted","Data":"b6f884388f26c61359bfd417c99b3e3db8ffd945e4b5639e7d7f5553b8d890cb"}
Jul 02 16:57:13.094186 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094162    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 16:57:13.094544 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094519    2581 generic.go:356] "Generic (PLEG): container finished" podID="51bec5ed-8842-47d8-8faa-9be7933368a9" containerID="25b5b190952fcef62fc1620a4200b998f3f63f5d8cb78cef8468832e7c1b726c" exitCode=1
Jul 02 16:57:13.094601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094580    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"64f627fd662f750d2878fb36dae5555bc9f3619d3da6ec3ad0bfa048a1c535b3"}
Jul 02 16:57:13.094679 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"4cc16755ad4e3b682520cf047e4dac0983053ae05eeb5ff749e73eac3d6bf24c"}
Jul 02 16:57:13.094736 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"d0900e41a055b085cc3260007e97766325a8bc1c9d0d3bf22fb01603414894d7"}
Jul 02 16:57:13.094794 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerDied","Data":"25b5b190952fcef62fc1620a4200b998f3f63f5d8cb78cef8468832e7c1b726c"}
Jul 02 16:57:13.094848 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.094821    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"1d8161b4a76d46f3586702a144f5aa0fc4f6ed383116f2273e36728238f767cc"}
Jul 02 16:57:13.096268 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.096248    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-s77nv" event={"ID":"c42e8c85-0da3-4dea-8cca-51ee7e3b781d","Type":"ContainerStarted","Data":"c2df72bfaae4e9533cf1f01c8ff758f2491a4381d94bb341e7ba8945756a0900"}
Jul 02 16:57:13.098907 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.098887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal" event={"ID":"e757ec6aacc35d99d6a9719bfafce01a","Type":"ContainerStarted","Data":"f07703cd66909a57ba65d67df68d9283829873989f9f771c54de855394ef01d0"}
Jul 02 16:57:13.101558 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.101522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-572tv" podStartSLOduration=3.311931255 podStartE2EDuration="24.101510564s" totalImagesPullingTime="20.789579309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:13.101016777 +0000 UTC m=+24.788017750" watchObservedRunningTime="2026-07-02 16:57:13.101510564 +0000 UTC m=+24.788511537"
Jul 02 16:57:13.119598 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:13.119522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-28.ec2.internal" podStartSLOduration=22.119505468 podStartE2EDuration="22.119505468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:13.119287775 +0000 UTC m=+24.806288767" watchObservedRunningTime="2026-07-02 16:57:13.119505468 +0000 UTC m=+24.806506473"
Jul 02 16:57:14.103535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.103506    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 16:57:14.104215 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.103879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"76e5cc7ea85b5932309b85250a9c1d18de41f324cca9e054105865cb5129e1ff"}
Jul 02 16:57:14.105176 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.105157    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-k8bnn" event={"ID":"9f23d5a2-8e77-4895-ab64-361cd53fbc90","Type":"ContainerStarted","Data":"3dda4c23ced75b501666d513491fa04ce9237cb30a7cb41aa281258695898075"}
Jul 02 16:57:14.106468 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.106443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp" event={"ID":"64d32f64-0814-4ab7-9d2a-ed25c1fee465","Type":"ContainerStarted","Data":"b2a08911afc918f87279e5fa491a5e19abb310e1450e6ab2c5bb234198b8ffd2"}
Jul 02 16:57:14.108610 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.108581    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wnmxk" event={"ID":"25be8fe2-195c-4ff1-ab86-cefa434f6eae","Type":"ContainerStarted","Data":"32787e1c1b5ef6d79d7e3603118a70d8c5ca33d4d754230608b79ca71e675089"}
Jul 02 16:57:14.111536 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.111513    2581 generic.go:356] "Generic (PLEG): container finished" podID="bd323a7b0d15c1fb472752ef1a5a873f" containerID="6794730b9158a0c69f18bc8701c99eaf61ec281ea1dbaf360749c1a4947054c1" exitCode=0
Jul 02 16:57:14.111615 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.111595    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal" event={"ID":"bd323a7b0d15c1fb472752ef1a5a873f","Type":"ContainerDied","Data":"6794730b9158a0c69f18bc8701c99eaf61ec281ea1dbaf360749c1a4947054c1"}
Jul 02 16:57:14.112986 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.112964    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="e76fb8a04bb9d0a2b3c79f0f2b25e2fa6e69a7adb3ed820042bf9f0ecac246bb" exitCode=0
Jul 02 16:57:14.113056 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.112992    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"e76fb8a04bb9d0a2b3c79f0f2b25e2fa6e69a7adb3ed820042bf9f0ecac246bb"}
Jul 02 16:57:14.114413 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.114395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g5t5n" event={"ID":"855cb017-d9f0-47da-800c-b325b3067bad","Type":"ContainerStarted","Data":"0e19035e6823b18e53cb7f7d5ebf18d379e1c6ba2ee7f92315f6c2700b804b6d"}
Jul 02 16:57:14.115598 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.115578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ct7pg" event={"ID":"5060ab2f-b2ce-4b34-8e08-2086e9af8e3c","Type":"ContainerStarted","Data":"a8b4bf9988f1b3196d89d62016f085364dfb4c3e4389cfa2e4061cc27433e1e7"}
Jul 02 16:57:14.122851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.122818    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-k8bnn" podStartSLOduration=3.3530205029999998 podStartE2EDuration="24.122808289s" totalImagesPullingTime="20.769787786s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:14.122580863 +0000 UTC m=+25.809581831" watchObservedRunningTime="2026-07-02 16:57:14.122808289 +0000 UTC m=+25.809809260"
Jul 02 16:57:14.123032 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.123012    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-s77nv" podStartSLOduration=4.307716463 podStartE2EDuration="25.123007887s" totalImagesPullingTime="20.815291424s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:13.144057634 +0000 UTC m=+24.831058606" watchObservedRunningTime="2026-07-02 16:57:14.123007887 +0000 UTC m=+25.810008858"
Jul 02 16:57:14.141099 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.141061    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g5t5n" podStartSLOduration=3.376000037 podStartE2EDuration="24.141047273s" totalImagesPullingTime="20.765047236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:14.140794462 +0000 UTC m=+25.827795434" watchObservedRunningTime="2026-07-02 16:57:14.141047273 +0000 UTC m=+25.828048245"
Jul 02 16:57:14.186069 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.186023    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ct7pg" podStartSLOduration=3.389920615 podStartE2EDuration="24.186008264s" totalImagesPullingTime="20.796087649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:14.185628221 +0000 UTC m=+25.872629185" watchObservedRunningTime="2026-07-02 16:57:14.186008264 +0000 UTC m=+25.873009262"
Jul 02 16:57:14.221068 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.221010    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wnmxk" podStartSLOduration=8.956105272 podStartE2EDuration="25.220991263s" totalImagesPullingTime="16.264885991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:14.220557498 +0000 UTC m=+25.907558470" watchObservedRunningTime="2026-07-02 16:57:14.220991263 +0000 UTC m=+25.907992236"
Jul 02 16:57:14.401046 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.400816    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 02 16:57:14.810584 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.810548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:14.810884 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.810546    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:14.810884 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.810686    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:14.810884 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:14.810685    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:14.810884 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:14.810784    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:14.811128 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:14.810891    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:14.843605 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.843502    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T16:57:14.401042989Z","UUID":"b4127962-b4dc-498f-9138-51cce10f9416","Handler":null,"Name":"","Endpoint":""}
Jul 02 16:57:14.845300 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.845278    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 02 16:57:14.845436 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:14.845317    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 02 16:57:15.119717 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:15.119621    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal" event={"ID":"bd323a7b0d15c1fb472752ef1a5a873f","Type":"ContainerStarted","Data":"c2864960e257cf99ba0c5e99809a38369dc55983c49426a50afed8df40a6c0c5"}
Jul 02 16:57:15.121680 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:15.121653    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp" event={"ID":"64d32f64-0814-4ab7-9d2a-ed25c1fee465","Type":"ContainerStarted","Data":"09bc1a2f45e1eda7d77d0a11f02906e250376fe713a3a7cb9dadae82ca465afe"}
Jul 02 16:57:15.137684 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:15.137636    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-28.ec2.internal" podStartSLOduration=24.137618898 podStartE2EDuration="24.137618898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:15.137490663 +0000 UTC m=+26.824491655" watchObservedRunningTime="2026-07-02 16:57:15.137618898 +0000 UTC m=+26.824619869"
Jul 02 16:57:16.128024 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.127952    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 16:57:16.128412 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.128354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"15297af7c80f855d1c69565e69626bbadcace4cb757d13c9d853742038fcdfbc"}
Jul 02 16:57:16.130601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.130560    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp" event={"ID":"64d32f64-0814-4ab7-9d2a-ed25c1fee465","Type":"ContainerStarted","Data":"6cfeeb5ccb7df69f16808dec87437279cfa889961514b95fb4dd7d7cbd6129e4"}
Jul 02 16:57:16.148400 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.148346    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vv8gp" podStartSLOduration=3.937694521 podStartE2EDuration="27.148326795s" totalImagesPullingTime="23.210632274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:16.147206101 +0000 UTC m=+27.834207112" watchObservedRunningTime="2026-07-02 16:57:16.148326795 +0000 UTC m=+27.835327769"
Jul 02 16:57:16.787054 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.787019    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:57:16.787787 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.787759    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:57:16.810479 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.810444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:16.810648 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:16.810564    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:16.810648 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.810441    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:16.810648 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:16.810444    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:16.810811 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:16.810663    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:16.810811 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:16.810750    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:17.133442 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:17.133409    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:57:17.133968 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:17.133939    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wnmxk"
Jul 02 16:57:18.810794 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:18.810595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:18.811458 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:18.810659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:18.811458 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:18.810891    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:18.811458 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:18.810682    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:18.811458 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:18.810956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:18.811458 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:18.811019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:19.139826 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.139745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 16:57:19.140125 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.140104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"eaeb986c51ce11876b5da8eb1a40e6b4550162a491875a7e4607f76e1eeffd0c"}
Jul 02 16:57:19.140365 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.140349    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:19.140436 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.140413    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:19.140568 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.140552    2581 scope.go:117] "RemoveContainer" containerID="25b5b190952fcef62fc1620a4200b998f3f63f5d8cb78cef8468832e7c1b726c"
Jul 02 16:57:19.142020 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.142000    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="5c4fdd3c1f7281f92589cd86de5a71519fb7c7823c43b324c4e4cf0abbeff14f" exitCode=0
Jul 02 16:57:19.142106 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.142073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"5c4fdd3c1f7281f92589cd86de5a71519fb7c7823c43b324c4e4cf0abbeff14f"}
Jul 02 16:57:19.156220 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:19.156200    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:20.148177 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.148153    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 16:57:20.148731 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.148566    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" event={"ID":"51bec5ed-8842-47d8-8faa-9be7933368a9","Type":"ContainerStarted","Data":"f2c4f43874dfab1c910f31a7253136ac4cdf4a0617a2c9d5e992930e80df71ee"}
Jul 02 16:57:20.148919 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.148900    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:20.163797 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.163767    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:20.186893 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.186813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx" podStartSLOduration=9.006086267 podStartE2EDuration="30.186798329s" totalImagesPullingTime="21.180712062s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:20.186430038 +0000 UTC m=+31.873431009" watchObservedRunningTime="2026-07-02 16:57:20.186798329 +0000 UTC m=+31.873799301"
Jul 02 16:57:20.558743 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.558710    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hx8nl"]
Jul 02 16:57:20.558919 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.558829    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:20.558972 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:20.558939    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:20.561968 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.561936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-srp4c"]
Jul 02 16:57:20.562107 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.562048    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:20.562174 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:20.562146    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:20.562678 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.562655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-565sb"]
Jul 02 16:57:20.562766 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:20.562750    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:20.562876 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:20.562840    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:21.152667 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:21.152633    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="4232efea89bb2c4bfe2ca3df6c0e50a0bf7a834b22834251d4b14cf50b25e971" exitCode=0
Jul 02 16:57:21.153089 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:21.152724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"4232efea89bb2c4bfe2ca3df6c0e50a0bf7a834b22834251d4b14cf50b25e971"}
Jul 02 16:57:21.810454 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:21.810418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:21.810617 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:21.810558    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:21.811000 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:21.810979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:21.811138 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:21.811078    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:22.703770 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:22.703519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:22.703770 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:22.703754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:22.704190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.703664    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 02 16:57:22.704190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.703835    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:57:22.704190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.703837    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret podName:a94f166e-7d6e-4716-9986-ea647fcb439c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:54.703823593 +0000 UTC m=+66.390824544 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret") pod "global-pull-secret-syncer-hx8nl" (UID: "a94f166e-7d6e-4716-9986-ea647fcb439c") : object "kube-system"/"original-pull-secret" not registered
Jul 02 16:57:22.704190 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.703903    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:54.703891367 +0000 UTC m=+66.390892317 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 02 16:57:22.804658 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:22.804622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:22.804809 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.804772    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 02 16:57:22.804809 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.804794    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 02 16:57:22.804809 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.804803    2581 projected.go:196] Error preparing data for projected volume kube-api-access-rrxwb for pod openshift-network-diagnostics/network-check-target-565sb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:57:22.804959 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.804851    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb podName:081c49cc-5d27-4e6d-ab39-765ed630394e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:54.80483774 +0000 UTC m=+66.491838691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rrxwb" (UniqueName: "kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb") pod "network-check-target-565sb" (UID: "081c49cc-5d27-4e6d-ab39-765ed630394e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 02 16:57:22.810227 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:22.810210    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:22.810327 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:22.810308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:23.159663 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:23.159631    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="97883cb1dc9f7621b75ed0c23853316b196e3e6dbda2f7f5febed1c985838405" exitCode=0
Jul 02 16:57:23.159822 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:23.159697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"97883cb1dc9f7621b75ed0c23853316b196e3e6dbda2f7f5febed1c985838405"}
Jul 02 16:57:23.810636 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:23.810599    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:23.811033 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:23.810637    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:23.811033 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:23.810808    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-srp4c" podUID="f22f8ef6-4542-4886-aed4-c4a7e8dcad53"
Jul 02 16:57:23.811033 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:23.810955    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-565sb" podUID="081c49cc-5d27-4e6d-ab39-765ed630394e"
Jul 02 16:57:24.810704 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:24.810673    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:24.811120 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:24.810799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hx8nl" podUID="a94f166e-7d6e-4716-9986-ea647fcb439c"
Jul 02 16:57:25.078517 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.078491    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-28.ec2.internal" event="NodeReady"
Jul 02 16:57:25.078671 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.078633    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 02 16:57:25.114541 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.114509    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:57:25.118328 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.118306    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.120918 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.120879    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lw9nb\""
Jul 02 16:57:25.120918 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.120895    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 02 16:57:25.121106 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.120954    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 02 16:57:25.121106 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.121016    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 02 16:57:25.127005 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.126981    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 02 16:57:25.131938 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.131911    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-q28mw"]
Jul 02 16:57:25.136184 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.136167    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:57:25.136284 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.136253    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.138949 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.138929    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vnppv"]
Jul 02 16:57:25.140195 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.140158    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-dockercfg-gz8mr\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gz8mr\"" type="*v1.Secret"
Jul 02 16:57:25.140971 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.140937    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"dns-default-metrics-tls\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret"
Jul 02 16:57:25.141627 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.141598    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"dns-default\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-dns\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap"
Jul 02 16:57:25.143887 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.143010    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.146082 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.146063    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 02 16:57:25.146267 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.146247    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 02 16:57:25.146357 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.146335    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 02 16:57:25.146831 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.146813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p65nb\""
Jul 02 16:57:25.165361 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.165111    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q28mw"]
Jul 02 16:57:25.165786 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.165763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnppv"]
Jul 02 16:57:25.225984 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.225949    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwjs\" (UniqueName: \"kubernetes.io/projected/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-kube-api-access-2rwjs\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.225984 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.225990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226203 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226203 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226118    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.226203 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-tmp-dir\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.226305 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226305 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226224    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226305 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226305 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226268    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppb9c\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226430 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226430 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226350    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.226430 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226364    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.226430 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226382    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1f9e1e-f388-449b-bffc-8b3096117108-cert\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.226430 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.226418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql2tx\" (UniqueName: \"kubernetes.io/projected/0b1f9e1e-f388-449b-bffc-8b3096117108-kube-api-access-ql2tx\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.327639 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.327605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1f9e1e-f388-449b-bffc-8b3096117108-cert\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.327639 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.327651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql2tx\" (UniqueName: \"kubernetes.io/projected/0b1f9e1e-f388-449b-bffc-8b3096117108-kube-api-access-ql2tx\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.327913 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.327880    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwjs\" (UniqueName: \"kubernetes.io/projected/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-kube-api-access-2rwjs\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.327982 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.327926    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.327982 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.327961    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328088 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328006    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.328088 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-tmp-dir\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.328183 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328183 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328127    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328183 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328183 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328178    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppb9c\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.328248    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.328265    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328268    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.328372 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.328319    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:25.828301959 +0000 UTC m=+37.515302925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:25.328700 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.328649    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.329586 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.329333    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-tmp-dir\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.329586 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.329442    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.329586 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.329547    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.332939 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.332896    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.332939 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.332920    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0b1f9e1e-f388-449b-bffc-8b3096117108-cert\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.333087 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.332966    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.333707 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.333663    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wnmxk_25be8fe2-195c-4ff1-ab86-cefa434f6eae/konnectivity-agent/0.log" limit=1024
Jul 02 16:57:25.337283 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.337262    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppb9c\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.337388 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.337316    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql2tx\" (UniqueName: \"kubernetes.io/projected/0b1f9e1e-f388-449b-bffc-8b3096117108-kube-api-access-ql2tx\") pod \"ingress-canary-vnppv\" (UID: \"0b1f9e1e-f388-449b-bffc-8b3096117108\") " pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.338110 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.338065    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwjs\" (UniqueName: \"kubernetes.io/projected/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-kube-api-access-2rwjs\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:25.338503 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.338464    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.422136 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.422099    2581 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wnmxk_25be8fe2-195c-4ff1-ab86-cefa434f6eae/konnectivity-agent/0.log" limit=1024
Jul 02 16:57:25.454334 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.454297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vnppv"
Jul 02 16:57:25.600259 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.600143    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vnppv"]
Jul 02 16:57:25.604489 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:25.604314    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1f9e1e_f388_449b_bffc_8b3096117108.slice/crio-c40f52b8c7fbe26aef9723348fad4e8cad9600ef81cb2be42085b33964c7f816 WatchSource:0}: Error finding container c40f52b8c7fbe26aef9723348fad4e8cad9600ef81cb2be42085b33964c7f816: Status 404 returned error can't find the container with id c40f52b8c7fbe26aef9723348fad4e8cad9600ef81cb2be42085b33964c7f816
Jul 02 16:57:25.810215 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.810182    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:25.810378 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.810183    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:25.813001 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.812979    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 16:57:25.813001 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.812999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwbzf\""
Jul 02 16:57:25.813419 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.812999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 16:57:25.813419 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.813091    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 16:57:25.813419 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.812999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xzcqc\""
Jul 02 16:57:25.832079 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:25.832056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:25.832218 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.832180    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:25.832218 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.832194    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:25.832311 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:25.832262    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:26.83224672 +0000 UTC m=+38.519247670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:26.063519 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.063488    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gz8mr\""
Jul 02 16:57:26.130496 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.130460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 02 16:57:26.138920 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.138891    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:26.139085 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.138969    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:26.638945077 +0000 UTC m=+38.325946029 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:26.169631 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.169593    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnppv" event={"ID":"0b1f9e1e-f388-449b-bffc-8b3096117108","Type":"ContainerStarted","Data":"c40f52b8c7fbe26aef9723348fad4e8cad9600ef81cb2be42085b33964c7f816"}
Jul 02 16:57:26.329015 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.328938    2581 configmap.go:193] Couldn't get configMap openshift-dns/dns-default: failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:26.329185 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.329037    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:26.829014036 +0000 UTC m=+38.516014986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:26.642551 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.642469    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:26.642710 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.642586    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:26.642710 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.642655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:27.642637227 +0000 UTC m=+39.329638195 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:26.741080 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.741042    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 02 16:57:26.810786 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.810752    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:26.813795 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.813702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 16:57:26.844183 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.844151    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:26.844301 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.844227    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:26.844344 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.844331    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:26.844382 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.844348    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:26.844441 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:26.844413    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:28.844391649 +0000 UTC m=+40.531392630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:26.844664 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:26.844647    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-config-volume\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:27.648639 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:27.648600    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:27.648815 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:27.648771    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:27.648928 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:27.648852    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:29.648830073 +0000 UTC m=+41.335831023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:28.177427 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:28.177390    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="eff95e7080f8f76a218a9f3b51c79e14424252ab1d0ea2635f2a97f9caa58623" exitCode=0
Jul 02 16:57:28.177936 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:28.177450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"eff95e7080f8f76a218a9f3b51c79e14424252ab1d0ea2635f2a97f9caa58623"}
Jul 02 16:57:28.858070 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:28.858035    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:28.858262 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:28.858169    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:28.858262 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:28.858190    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:28.858374 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:28.858280    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:32.858258231 +0000 UTC m=+44.545259181 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:29.183254 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:29.183165    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b0f716e-c53d-401c-8f26-e65f68a94306" containerID="c2ad987c821021bd0ffd579aa9121edda19f8aa4d15c4a63ee72f06308d438e9" exitCode=0
Jul 02 16:57:29.183254 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:29.183217    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerDied","Data":"c2ad987c821021bd0ffd579aa9121edda19f8aa4d15c4a63ee72f06308d438e9"}
Jul 02 16:57:29.665064 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:29.665024    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:29.665202 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:29.665184    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:29.665258 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:29.665247    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:33.665232582 +0000 UTC m=+45.352233531 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:30.186773 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:30.186732    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vnppv" event={"ID":"0b1f9e1e-f388-449b-bffc-8b3096117108","Type":"ContainerStarted","Data":"17cd7229893ed13e61407ad4e01dc717ced9eb03824f23a844effba6d105d477"}
Jul 02 16:57:30.189821 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:30.189794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" event={"ID":"9b0f716e-c53d-401c-8f26-e65f68a94306","Type":"ContainerStarted","Data":"aff979ed0d2fa299d8cbdbc2443ffcde1fb0f0eb643fd4f88c32370687661459"}
Jul 02 16:57:30.202194 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:30.202153    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vnppv" podStartSLOduration=1.089318428 podStartE2EDuration="5.202139438s" totalImagesPullingTime="4.11282101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:30.201245305 +0000 UTC m=+41.888246303" watchObservedRunningTime="2026-07-02 16:57:30.202139438 +0000 UTC m=+41.889140421"
Jul 02 16:57:30.225419 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:30.225318    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9s2wn" podStartSLOduration=9.190260962 podStartE2EDuration="40.225295666s" totalImagesPullingTime="31.035034704s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:30.2243068 +0000 UTC m=+41.911307786" watchObservedRunningTime="2026-07-02 16:57:30.225295666 +0000 UTC m=+41.912296639"
Jul 02 16:57:32.888407 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:32.888365    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:32.888775 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:32.888513    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:32.888775 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:32.888530    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:32.888775 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:32.888586    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:40.888571039 +0000 UTC m=+52.575571988 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:33.693469 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:33.693429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:33.693652 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:33.693577    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:33.693652 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:33.693645    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:41.693629032 +0000 UTC m=+53.380629987 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:40.946383 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:40.946344    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:40.946844 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:40.946464    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:40.946844 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:40.946475    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:40.946844 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:40.946527    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:57:56.946512602 +0000 UTC m=+68.633513552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:41.753072 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:41.753033    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:41.753242 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:41.753148    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:41.753242 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:41.753199    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:57.753186532 +0000 UTC m=+69.440187486 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:48.432221 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.432184    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5f67b55555-bj6rc"]
Jul 02 16:57:48.434867 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.434842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.439304 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.439282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 02 16:57:48.440161 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.440143    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qvl8t\""
Jul 02 16:57:48.440521 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.440505    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 02 16:57:48.440521 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.440515    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 02 16:57:48.447887 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.447852    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 02 16:57:48.451108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.451091    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 02 16:57:48.463211 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.463157    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-bj6rc"]
Jul 02 16:57:48.499390 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499361    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hj7q\" (UniqueName: \"kubernetes.io/projected/54d70273-357a-456a-86aa-8f945e48f656-kube-api-access-4hj7q\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.499528 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499402    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-snapshots\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.499528 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499493    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-tmp\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.499528 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d70273-357a-456a-86aa-8f945e48f656-serving-cert\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.499642 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499545    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.499642 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.499559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-service-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.509294 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.509266    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"]
Jul 02 16:57:48.511123 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.511109    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.515242 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.515221    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:57:48.515892 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.515848    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 02 16:57:48.515892 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.515880    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"serving-cert\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" type="*v1.Secret"
Jul 02 16:57:48.516053 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.516022    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"service-ca-operator-5d7b7dc95c-7f7ld\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="6f2f11c5-ab72-4155-aa31-5c32c32d89cc" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.516226 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.516197    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"service-ca-operator-config\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" type="*v1.ConfigMap"
Jul 02 16:57:48.516342 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.516317    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"service-ca-operator-dockercfg-m8zxj\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-service-ca-operator\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m8zxj\"" type="*v1.Secret"
Jul 02 16:57:48.543567 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.543543    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"]
Jul 02 16:57:48.545675 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.545661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.553221 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.553205    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 16:57:48.553311 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.553209    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 02 16:57:48.553348 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.553306    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2mrpm\""
Jul 02 16:57:48.553767 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.553752    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:57:48.559928 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.559911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"]
Jul 02 16:57:48.600329 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600329 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-service-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600542 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600353    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2rpw\" (UniqueName: \"kubernetes.io/projected/0f01d317-65e0-4da9-a285-bff0bbcc317e-kube-api-access-j2rpw\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.600542 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600387    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hj7q\" (UniqueName: \"kubernetes.io/projected/54d70273-357a-456a-86aa-8f945e48f656-kube-api-access-4hj7q\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600542 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600490    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnlf5\" (UniqueName: \"kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.600542 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.600744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600553    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.600744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-snapshots\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-tmp\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d70273-357a-456a-86aa-8f945e48f656-serving-cert\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.600744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600733    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.601024 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.600907    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-service-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.601185 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.601164    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-tmp\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.601253 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.601225    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/54d70273-357a-456a-86aa-8f945e48f656-snapshots\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.601253 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.601235    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54d70273-357a-456a-86aa-8f945e48f656-trusted-ca-bundle\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.603341 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.603320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-gptgl"]
Jul 02 16:57:48.603945 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.603924    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/54d70273-357a-456a-86aa-8f945e48f656-serving-cert\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.605732 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.605715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl"
Jul 02 16:57:48.616851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.616835    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bvdsj\""
Jul 02 16:57:48.620353 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.620332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"]
Jul 02 16:57:48.621041 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.621010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-gptgl"]
Jul 02 16:57:48.627815 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.627793    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hj7q\" (UniqueName: \"kubernetes.io/projected/54d70273-357a-456a-86aa-8f945e48f656-kube-api-access-4hj7q\") pod \"insights-operator-5f67b55555-bj6rc\" (UID: \"54d70273-357a-456a-86aa-8f945e48f656\") " pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.701654 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.701654 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701620    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2rpw\" (UniqueName: \"kubernetes.io/projected/0f01d317-65e0-4da9-a285-bff0bbcc317e-kube-api-access-j2rpw\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.701654 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlf5\" (UniqueName: \"kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.701851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:48.701851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.701851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.701709    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8hzs\" (UniqueName: \"kubernetes.io/projected/da12c6fb-fc38-4f96-b573-94027aef51ea-kube-api-access-g8hzs\") pod \"network-check-source-56499946bc-gptgl\" (UID: \"da12c6fb-fc38-4f96-b573-94027aef51ea\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl"
Jul 02 16:57:48.701851 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.701801    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 16:57:48.702068 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:48.701876    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls podName:0f01d317-65e0-4da9-a285-bff0bbcc317e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:49.201838131 +0000 UTC m=+60.888839083 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9g2t6" (UID: "0f01d317-65e0-4da9-a285-bff0bbcc317e") : secret "samples-operator-tls" not found
Jul 02 16:57:48.721346 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.721323    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 02 16:57:48.732520 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.732501    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:57:48.743047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.743026    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2rpw\" (UniqueName: \"kubernetes.io/projected/0f01d317-65e0-4da9-a285-bff0bbcc317e-kube-api-access-j2rpw\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:48.747130 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.747113    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-qvl8t\""
Jul 02 16:57:48.754810 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.754788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5f67b55555-bj6rc"
Jul 02 16:57:48.802661 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.802478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8hzs\" (UniqueName: \"kubernetes.io/projected/da12c6fb-fc38-4f96-b573-94027aef51ea-kube-api-access-g8hzs\") pod \"network-check-source-56499946bc-gptgl\" (UID: \"da12c6fb-fc38-4f96-b573-94027aef51ea\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl"
Jul 02 16:57:48.825229 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.825200    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 02 16:57:48.832942 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.832921    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 02 16:57:48.862876 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.859809    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8hzs\" (UniqueName: \"kubernetes.io/projected/da12c6fb-fc38-4f96-b573-94027aef51ea-kube-api-access-g8hzs\") pod \"network-check-source-56499946bc-gptgl\" (UID: \"da12c6fb-fc38-4f96-b573-94027aef51ea\") " pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl"
Jul 02 16:57:48.904052 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.904023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5f67b55555-bj6rc"]
Jul 02 16:57:48.907091 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:48.907066    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54d70273_357a_456a_86aa_8f945e48f656.slice/crio-52c790bcb3caaab77054b9d152e681c8512a7a50d500864791261296208cdc7f WatchSource:0}: Error finding container 52c790bcb3caaab77054b9d152e681c8512a7a50d500864791261296208cdc7f: Status 404 returned error can't find the container with id 52c790bcb3caaab77054b9d152e681c8512a7a50d500864791261296208cdc7f
Jul 02 16:57:48.917995 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.917974    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-bvdsj\""
Jul 02 16:57:48.925555 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:48.925536    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl"
Jul 02 16:57:49.050577 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.050549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-56499946bc-gptgl"]
Jul 02 16:57:49.053934 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:49.053908    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda12c6fb_fc38_4f96_b573_94027aef51ea.slice/crio-ff94f2b909b52ab6b2ee4dea03ae01300cc1d3dceed12c6a7dbde7eaeb8cad55 WatchSource:0}: Error finding container ff94f2b909b52ab6b2ee4dea03ae01300cc1d3dceed12c6a7dbde7eaeb8cad55: Status 404 returned error can't find the container with id ff94f2b909b52ab6b2ee4dea03ae01300cc1d3dceed12c6a7dbde7eaeb8cad55
Jul 02 16:57:49.205655 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.205619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:49.205817 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.205761    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 16:57:49.205919 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.205824    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls podName:0f01d317-65e0-4da9-a285-bff0bbcc317e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:50.205807526 +0000 UTC m=+61.892808479 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9g2t6" (UID: "0f01d317-65e0-4da9-a285-bff0bbcc317e") : secret "samples-operator-tls" not found
Jul 02 16:57:49.233363 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.233288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl" event={"ID":"da12c6fb-fc38-4f96-b573-94027aef51ea","Type":"ContainerStarted","Data":"ff94f2b909b52ab6b2ee4dea03ae01300cc1d3dceed12c6a7dbde7eaeb8cad55"}
Jul 02 16:57:49.234333 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.234309    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-bj6rc" event={"ID":"54d70273-357a-456a-86aa-8f945e48f656","Type":"ContainerStarted","Data":"52c790bcb3caaab77054b9d152e681c8512a7a50d500864791261296208cdc7f"}
Jul 02 16:57:49.686278 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.686242    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-m8zxj\""
Jul 02 16:57:49.702766 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.702736    2581 configmap.go:193] Couldn't get configMap openshift-service-ca-operator/service-ca-operator-config: failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:49.702953 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.702821    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config podName:6f2f11c5-ab72-4155-aa31-5c32c32d89cc nodeName:}" failed. No retries permitted until 2026-07-02 16:57:50.202799501 +0000 UTC m=+61.889800456 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config" (UniqueName: "kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config") pod "service-ca-operator-5d7b7dc95c-7f7ld" (UID: "6f2f11c5-ab72-4155-aa31-5c32c32d89cc") : failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:49.702953 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.702737    2581 secret.go:189] Couldn't get secret openshift-service-ca-operator/serving-cert: failed to sync secret cache: timed out waiting for the condition
Jul 02 16:57:49.702953 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.702898    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert podName:6f2f11c5-ab72-4155-aa31-5c32c32d89cc nodeName:}" failed. No retries permitted until 2026-07-02 16:57:50.202879028 +0000 UTC m=+61.889879992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "serving-cert" (UniqueName: "kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert") pod "service-ca-operator-5d7b7dc95c-7f7ld" (UID: "6f2f11c5-ab72-4155-aa31-5c32c32d89cc") : failed to sync secret cache: timed out waiting for the condition
Jul 02 16:57:49.717095 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.717065    2581 projected.go:291] Couldn't get configMap openshift-service-ca-operator/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:49.720407 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.720385    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 02 16:57:49.727264 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.727240    2581 projected.go:196] Error preparing data for projected volume kube-api-access-xnlf5 for pod openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld: failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:49.727392 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:49.727314    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5 podName:6f2f11c5-ab72-4155-aa31-5c32c32d89cc nodeName:}" failed. No retries permitted until 2026-07-02 16:57:50.22729474 +0000 UTC m=+61.914295707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xnlf5" (UniqueName: "kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5") pod "service-ca-operator-5d7b7dc95c-7f7ld" (UID: "6f2f11c5-ab72-4155-aa31-5c32c32d89cc") : failed to sync configmap cache: timed out waiting for the condition
Jul 02 16:57:49.918947 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:49.918913    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 02 16:57:50.056342 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.056314    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 02 16:57:50.069367 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.069342    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 02 16:57:50.215161 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.215129    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.215350 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.215169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:50.215350 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:50.215291    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 16:57:50.215350 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:50.215349    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls podName:0f01d317-65e0-4da9-a285-bff0bbcc317e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:52.215330887 +0000 UTC m=+63.902331836 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9g2t6" (UID: "0f01d317-65e0-4da9-a285-bff0bbcc317e") : secret "samples-operator-tls" not found
Jul 02 16:57:50.215500 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.215427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.216050 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.216028    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-config\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.217923 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.217899    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-serving-cert\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.316399 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.316313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnlf5\" (UniqueName: \"kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.319298 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.319269    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnlf5\" (UniqueName: \"kubernetes.io/projected/6f2f11c5-ab72-4155-aa31-5c32c32d89cc-kube-api-access-xnlf5\") pod \"service-ca-operator-5d7b7dc95c-7f7ld\" (UID: \"6f2f11c5-ab72-4155-aa31-5c32c32d89cc\") " pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.321025 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.320999    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"
Jul 02 16:57:50.463544 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:50.463514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld"]
Jul 02 16:57:50.466840 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:50.466816    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2f11c5_ab72_4155_aa31_5c32c32d89cc.slice/crio-3d089d784aaed3fd538d17054416e33994f8d6bdfe880289dcffc43aa7728981 WatchSource:0}: Error finding container 3d089d784aaed3fd538d17054416e33994f8d6bdfe880289dcffc43aa7728981: Status 404 returned error can't find the container with id 3d089d784aaed3fd538d17054416e33994f8d6bdfe880289dcffc43aa7728981
Jul 02 16:57:51.241722 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:51.241675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld" event={"ID":"6f2f11c5-ab72-4155-aa31-5c32c32d89cc","Type":"ContainerStarted","Data":"3d089d784aaed3fd538d17054416e33994f8d6bdfe880289dcffc43aa7728981"}
Jul 02 16:57:52.176312 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:52.176280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x9mwx"
Jul 02 16:57:52.233438 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:52.233402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:52.233623 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:52.233527    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 16:57:52.233623 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:52.233598    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls podName:0f01d317-65e0-4da9-a285-bff0bbcc317e nodeName:}" failed. No retries permitted until 2026-07-02 16:57:56.233575346 +0000 UTC m=+67.920576310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9g2t6" (UID: "0f01d317-65e0-4da9-a285-bff0bbcc317e") : secret "samples-operator-tls" not found
Jul 02 16:57:53.248747 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.248708    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl" event={"ID":"da12c6fb-fc38-4f96-b573-94027aef51ea","Type":"ContainerStarted","Data":"bc6f991bcc7d4ac320135b07c795ff0bfb477f1443c5f2b3e6b272f7432459f0"}
Jul 02 16:57:53.250688 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.250650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld" event={"ID":"6f2f11c5-ab72-4155-aa31-5c32c32d89cc","Type":"ContainerStarted","Data":"2720df380537d7a08efe6553ec3df819660a91ae14c4d2ff95d1f4a72cbd7446"}
Jul 02 16:57:53.252217 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.252197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-bj6rc" event={"ID":"54d70273-357a-456a-86aa-8f945e48f656","Type":"ContainerStarted","Data":"ab583f14c8f8c32300b2c710b87c650a4b28aa838c4ebcd5a4375374c6bc7d4c"}
Jul 02 16:57:53.289655 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.289548    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-56499946bc-gptgl" podStartSLOduration=1.287749232 podStartE2EDuration="5.289529836s" totalImagesPullingTime="4.001780604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:53.289398332 +0000 UTC m=+64.976399306" watchObservedRunningTime="2026-07-02 16:57:53.289529836 +0000 UTC m=+64.976530811"
Jul 02 16:57:53.399261 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.399127    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld" podStartSLOduration=2.811513862 podStartE2EDuration="5.399103024s" totalImagesPullingTime="2.587589162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:53.339331016 +0000 UTC m=+65.026331988" watchObservedRunningTime="2026-07-02 16:57:53.399103024 +0000 UTC m=+65.086103998"
Jul 02 16:57:53.399744 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.399713    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5f67b55555-bj6rc" podStartSLOduration=1.259497303 podStartE2EDuration="5.399700038s" totalImagesPullingTime="4.140202735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:53.398945405 +0000 UTC m=+65.085946378" watchObservedRunningTime="2026-07-02 16:57:53.399700038 +0000 UTC m=+65.086701012"
Jul 02 16:57:53.621523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.621491    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"]
Jul 02 16:57:53.624715 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.624694    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"
Jul 02 16:57:53.630844 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.630822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 02 16:57:53.632372 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.632348    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 02 16:57:53.640596 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.640579    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-glck2\""
Jul 02 16:57:53.681967 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.681937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"]
Jul 02 16:57:53.749787 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.749751    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2rql\" (UniqueName: \"kubernetes.io/projected/2cfd618b-02ee-455f-90da-44d33a7b58a5-kube-api-access-c2rql\") pod \"migrator-786f9fbc5-wxjkf\" (UID: \"2cfd618b-02ee-455f-90da-44d33a7b58a5\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"
Jul 02 16:57:53.851249 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.851213    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2rql\" (UniqueName: \"kubernetes.io/projected/2cfd618b-02ee-455f-90da-44d33a7b58a5-kube-api-access-c2rql\") pod \"migrator-786f9fbc5-wxjkf\" (UID: \"2cfd618b-02ee-455f-90da-44d33a7b58a5\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"
Jul 02 16:57:53.878872 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.878777    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2rql\" (UniqueName: \"kubernetes.io/projected/2cfd618b-02ee-455f-90da-44d33a7b58a5-kube-api-access-c2rql\") pod \"migrator-786f9fbc5-wxjkf\" (UID: \"2cfd618b-02ee-455f-90da-44d33a7b58a5\") " pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"
Jul 02 16:57:53.934979 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:53.934939    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"
Jul 02 16:57:54.083765 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.083736    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf"]
Jul 02 16:57:54.086773 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:54.086746    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cfd618b_02ee_455f_90da_44d33a7b58a5.slice/crio-81506baaf9050ec2af1e9148cbe86e97c765c98db571c25cfe0eedce6fe79ee4 WatchSource:0}: Error finding container 81506baaf9050ec2af1e9148cbe86e97c765c98db571c25cfe0eedce6fe79ee4: Status 404 returned error can't find the container with id 81506baaf9050ec2af1e9148cbe86e97c765c98db571c25cfe0eedce6fe79ee4
Jul 02 16:57:54.256913 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.256807    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf" event={"ID":"2cfd618b-02ee-455f-90da-44d33a7b58a5","Type":"ContainerStarted","Data":"81506baaf9050ec2af1e9148cbe86e97c765c98db571c25cfe0eedce6fe79ee4"}
Jul 02 16:57:54.455429 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.455393    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dhb94"]
Jul 02 16:57:54.459847 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.459818    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.463507 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.463471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-4pq6p\""
Jul 02 16:57:54.463730 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.463709    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 02 16:57:54.464587 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.464568    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 02 16:57:54.493044 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.493013    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhb94"]
Jul 02 16:57:54.557756 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.557718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9a39120-edd8-43ac-b7ff-016eee8f0658-crio-socket\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.557983 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.557764    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a39120-edd8-43ac-b7ff-016eee8f0658-data-volume\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.557983 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.557880    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9g4h\" (UniqueName: \"kubernetes.io/projected/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-api-access-b9g4h\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.557983 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.557969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.558110 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.558022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659270 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9a39120-edd8-43ac-b7ff-016eee8f0658-crio-socket\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659270 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659269    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a39120-edd8-43ac-b7ff-016eee8f0658-data-volume\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659556 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9g4h\" (UniqueName: \"kubernetes.io/projected/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-api-access-b9g4h\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659556 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659367    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b9a39120-edd8-43ac-b7ff-016eee8f0658-crio-socket\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659556 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659390    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659556 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659430    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.659556 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.659497    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 16:57:54.659816 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.659575    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:55.159552223 +0000 UTC m=+66.846553190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : configmap "kube-rbac-proxy" not found
Jul 02 16:57:54.659816 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.659587    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:54.659816 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.659635    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:55.159620211 +0000 UTC m=+66.846621160 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:54.659816 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.659683    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b9a39120-edd8-43ac-b7ff-016eee8f0658-data-volume\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.684453 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.684423    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9g4h\" (UniqueName: \"kubernetes.io/projected/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-api-access-b9g4h\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:54.760239 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.760193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:57:54.760387 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.760290    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:54.763694 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.763606    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 02 16:57:54.763832 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.763722    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 02 16:57:54.770883 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.770834    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 02 16:57:54.771018 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:54.770929    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs podName:f22f8ef6-4542-4886-aed4-c4a7e8dcad53 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:58.770911392 +0000 UTC m=+130.457912344 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs") pod "network-metrics-daemon-srp4c" (UID: "f22f8ef6-4542-4886-aed4-c4a7e8dcad53") : secret "metrics-daemon-secret" not found
Jul 02 16:57:54.772982 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.772956    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a94f166e-7d6e-4716-9986-ea647fcb439c-original-pull-secret\") pod \"global-pull-secret-syncer-hx8nl\" (UID: \"a94f166e-7d6e-4716-9986-ea647fcb439c\") " pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:54.861486 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.861401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:54.864235 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.864210    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrxwb\" (UniqueName: \"kubernetes.io/projected/081c49cc-5d27-4e6d-ab39-765ed630394e-kube-api-access-rrxwb\") pod \"network-check-target-565sb\" (UID: \"081c49cc-5d27-4e6d-ab39-765ed630394e\") " pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:54.926149 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.926115    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwbzf\""
Jul 02 16:57:54.933843 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:54.933816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:55.021168 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.021139    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hx8nl"
Jul 02 16:57:55.088077 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.087707    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-565sb"]
Jul 02 16:57:55.165371 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.165270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:55.165371 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.165328    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:55.165586 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:55.165425    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 16:57:55.165586 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:55.165469    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:55.165586 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:55.165506    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:56.165484124 +0000 UTC m=+67.852485098 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : configmap "kube-rbac-proxy" not found
Jul 02 16:57:55.165586 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:55.165536    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:56.165516149 +0000 UTC m=+67.852517101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:55.200123 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:55.200090    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod081c49cc_5d27_4e6d_ab39_765ed630394e.slice/crio-dd35e299af92f1a60d83c94fdebefe5aa939b1ec593ac1b1ea45403360623eaf WatchSource:0}: Error finding container dd35e299af92f1a60d83c94fdebefe5aa939b1ec593ac1b1ea45403360623eaf: Status 404 returned error can't find the container with id dd35e299af92f1a60d83c94fdebefe5aa939b1ec593ac1b1ea45403360623eaf
Jul 02 16:57:55.261279 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.261250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-565sb" event={"ID":"081c49cc-5d27-4e6d-ab39-765ed630394e","Type":"ContainerStarted","Data":"dd35e299af92f1a60d83c94fdebefe5aa939b1ec593ac1b1ea45403360623eaf"}
Jul 02 16:57:55.345892 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:55.345852    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hx8nl"]
Jul 02 16:57:55.355896 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:57:55.355433    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94f166e_7d6e_4716_9986_ea647fcb439c.slice/crio-bf08db0eea94a28b5bfac547b5a2656f99b67f95e3570c2de748b80eb09ba76b WatchSource:0}: Error finding container bf08db0eea94a28b5bfac547b5a2656f99b67f95e3570c2de748b80eb09ba76b: Status 404 returned error can't find the container with id bf08db0eea94a28b5bfac547b5a2656f99b67f95e3570c2de748b80eb09ba76b
Jul 02 16:57:56.172188 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.172149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:56.172188 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.172192    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:56.172386 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.172277    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 16:57:56.172386 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.172293    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:56.172386 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.172343    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:58.172330903 +0000 UTC m=+69.859331857 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:56.172386 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.172356    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:57:58.172350256 +0000 UTC m=+69.859351206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : configmap "kube-rbac-proxy" not found
Jul 02 16:57:56.268335 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.268236    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf" event={"ID":"2cfd618b-02ee-455f-90da-44d33a7b58a5","Type":"ContainerStarted","Data":"c5ef3e5d1169131badb3eae484b28ca2a48b92daf05e6f42a5f72251a0e5581d"}
Jul 02 16:57:56.268335 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.268300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf" event={"ID":"2cfd618b-02ee-455f-90da-44d33a7b58a5","Type":"ContainerStarted","Data":"5815fff835784c40350312b0799361e946cfde28d114b923dab2f712f1e15566"}
Jul 02 16:57:56.269502 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.269469    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hx8nl" event={"ID":"a94f166e-7d6e-4716-9986-ea647fcb439c","Type":"ContainerStarted","Data":"bf08db0eea94a28b5bfac547b5a2656f99b67f95e3570c2de748b80eb09ba76b"}
Jul 02 16:57:56.271020 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.270999    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-565sb" event={"ID":"081c49cc-5d27-4e6d-ab39-765ed630394e","Type":"ContainerStarted","Data":"8b5b75f5ae800b9f97600154fc23aa951d31857bca0267dadbc0ce072a33472d"}
Jul 02 16:57:56.271135 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.271122    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:57:56.273640 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.273616    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:57:56.273757 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.273740    2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 02 16:57:56.273823 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.273813    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls podName:0f01d317-65e0-4da9-a285-bff0bbcc317e nodeName:}" failed. No retries permitted until 2026-07-02 16:58:04.273792165 +0000 UTC m=+75.960793127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls") pod "cluster-samples-operator-9ddc5ff76-9g2t6" (UID: "0f01d317-65e0-4da9-a285-bff0bbcc317e") : secret "samples-operator-tls" not found
Jul 02 16:57:56.297360 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.297300    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-786f9fbc5-wxjkf" podStartSLOduration=2.128557495 podStartE2EDuration="3.297280078s" totalImagesPullingTime="1.168722583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:56.296356499 +0000 UTC m=+67.983357497" watchObservedRunningTime="2026-07-02 16:57:56.297280078 +0000 UTC m=+67.984281051"
Jul 02 16:57:56.323948 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.323892    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-565sb" podStartSLOduration=66.323870701 podStartE2EDuration="1m6.323870701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:57:56.323868067 +0000 UTC m=+68.010869057" watchObservedRunningTime="2026-07-02 16:57:56.323870701 +0000 UTC m=+68.010871668"
Jul 02 16:57:56.976500 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.976469    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8bnn_9f23d5a2-8e77-4895-ab64-361cd53fbc90/dns-node-resolver/0.log"
Jul 02 16:57:56.981872 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:56.981826    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:57:56.982004 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.981979    2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 02 16:57:56.982004 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.981994    2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5dd8974cf-tlvxg: secret "image-registry-tls" not found
Jul 02 16:57:56.982087 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:56.982055    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls podName:c77a624a-c496-44bf-a03f-10a1c5e38c7c nodeName:}" failed. No retries permitted until 2026-07-02 16:58:28.982036989 +0000 UTC m=+100.669037942 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls") pod "image-registry-5dd8974cf-tlvxg" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c") : secret "image-registry-tls" not found
Jul 02 16:57:57.584778 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:57.584719    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g5t5n_855cb017-d9f0-47da-800c-b325b3067bad/node-ca/0.log"
Jul 02 16:57:57.790181 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:57.790149    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:57:57.790335 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:57.790300    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 02 16:57:57.790386 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:57.790361    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls podName:dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:29.790345964 +0000 UTC m=+101.477346926 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls") pod "dns-default-q28mw" (UID: "dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8") : secret "dns-default-metrics-tls" not found
Jul 02 16:57:58.192650 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:58.192609    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:58.192852 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:58.192786    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:58.192852 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:58.192841    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 16:57:58.192852 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:58.192853    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:02.19283763 +0000 UTC m=+73.879838580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:57:58.192852 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:58.192788    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:57:58.193107 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:57:58.192906    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:02.192889202 +0000 UTC m=+73.879890155 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : configmap "kube-rbac-proxy" not found
Jul 02 16:57:58.581569 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:58.581511    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnppv_0b1f9e1e-f388-449b-bffc-8b3096117108/serve-healthcheck-canary/0.log"
Jul 02 16:57:58.776205 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:58.776159    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-wxjkf_2cfd618b-02ee-455f-90da-44d33a7b58a5/migrator/0.log"
Jul 02 16:57:58.987480 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:57:58.987388    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-wxjkf_2cfd618b-02ee-455f-90da-44d33a7b58a5/graceful-termination/0.log"
Jul 02 16:58:00.286968 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:00.286882    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hx8nl" event={"ID":"a94f166e-7d6e-4716-9986-ea647fcb439c","Type":"ContainerStarted","Data":"d88b253a98b04e1348e381abd1a023d182d9d183c5375309ec177ed648368508"}
Jul 02 16:58:00.327106 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:00.327060    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hx8nl" podStartSLOduration=65.821714148 podStartE2EDuration="1m10.327045127s" totalImagesPullingTime="4.505330979s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:00.323745654 +0000 UTC m=+72.010746625" watchObservedRunningTime="2026-07-02 16:58:00.327045127 +0000 UTC m=+72.014046098"
Jul 02 16:58:02.224308 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:02.224259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:02.224308 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:02.224314    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:02.224821 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:02.224388    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 02 16:58:02.224821 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:02.224459    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:10.22444384 +0000 UTC m=+81.911444789 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : configmap "kube-rbac-proxy" not found
Jul 02 16:58:02.224821 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:02.224505    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:58:02.224821 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:02.224571    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:10.224553072 +0000 UTC m=+81.911554022 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:58:04.341766 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:04.341729    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:58:04.344410 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:04.344383    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0f01d317-65e0-4da9-a285-bff0bbcc317e-samples-operator-tls\") pod \"cluster-samples-operator-9ddc5ff76-9g2t6\" (UID: \"0f01d317-65e0-4da9-a285-bff0bbcc317e\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:58:04.457537 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:04.457502    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2mrpm\""
Jul 02 16:58:04.464394 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:04.464366    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"
Jul 02 16:58:04.603517 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:04.603493    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6"]
Jul 02 16:58:05.303893 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:05.303841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6" event={"ID":"0f01d317-65e0-4da9-a285-bff0bbcc317e","Type":"ContainerStarted","Data":"701c1ba745951ff15ba6dca0d076260a3eb9013ee61b0bcfa93cac33265f824f"}
Jul 02 16:58:06.309298 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:06.309266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6" event={"ID":"0f01d317-65e0-4da9-a285-bff0bbcc317e","Type":"ContainerStarted","Data":"1633a58d38aa01a7a73a60f8c1d17934b1007e09fee3e454c910df7f3023c1ce"}
Jul 02 16:58:06.309710 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:06.309318    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6" event={"ID":"0f01d317-65e0-4da9-a285-bff0bbcc317e","Type":"ContainerStarted","Data":"a29eccd5546e6ec6bdd1d9e0581b47f9f522a37b54c1550c8e1c858ec916ea8f"}
Jul 02 16:58:10.290327 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:10.290283    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:10.290327 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:10.290333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:10.290777 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:10.290458    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 02 16:58:10.290777 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:10.290509    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls podName:b9a39120-edd8-43ac-b7ff-016eee8f0658 nodeName:}" failed. No retries permitted until 2026-07-02 16:58:26.290496319 +0000 UTC m=+97.977497274 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dhb94" (UID: "b9a39120-edd8-43ac-b7ff-016eee8f0658") : secret "insights-runtime-extractor-tls" not found
Jul 02 16:58:10.290851 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:10.290792    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b9a39120-edd8-43ac-b7ff-016eee8f0658-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:26.312751 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.312606    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:26.314963 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.314940    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b9a39120-edd8-43ac-b7ff-016eee8f0658-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dhb94\" (UID: \"b9a39120-edd8-43ac-b7ff-016eee8f0658\") " pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:26.571342 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.571255    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dhb94"
Jul 02 16:58:26.710806 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.710634    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-9ddc5ff76-9g2t6" podStartSLOduration=37.354601059 podStartE2EDuration="38.710612084s" totalImagesPullingTime="1.356011025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:06.334071611 +0000 UTC m=+78.021072583" watchObservedRunningTime="2026-07-02 16:58:26.710612084 +0000 UTC m=+98.397613058"
Jul 02 16:58:26.710893 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.710869    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dhb94"]
Jul 02 16:58:26.713424 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:26.713398    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9a39120_edd8_43ac_b7ff_016eee8f0658.slice/crio-89c4ae93753364fc7ca3c264a0408ab5926caa83a491d2b2a6976f8412fbc12f WatchSource:0}: Error finding container 89c4ae93753364fc7ca3c264a0408ab5926caa83a491d2b2a6976f8412fbc12f: Status 404 returned error can't find the container with id 89c4ae93753364fc7ca3c264a0408ab5926caa83a491d2b2a6976f8412fbc12f
Jul 02 16:58:26.839557 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.839484    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"]
Jul 02 16:58:26.843660 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.843645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:26.846593 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.846567    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-ws5ff\""
Jul 02 16:58:26.847219 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.847076    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 02 16:58:26.847919 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.847895    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 02 16:58:26.848845 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.848828    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"]
Jul 02 16:58:26.851934 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.851914    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:26.852270 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.852248    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"]
Jul 02 16:58:26.855121 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.855105    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:26.855580 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.855559    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 02 16:58:26.855656 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.855597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 02 16:58:26.855981 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.855963    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 02 16:58:26.855981 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.855977    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 02 16:58:26.864059 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.864044    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 02 16:58:26.864463 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.864416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 02 16:58:26.864463 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.864426    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 02 16:58:26.867721 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.867702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"]
Jul 02 16:58:26.869761 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.869744    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 02 16:58:26.879450 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.879431    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"]
Jul 02 16:58:26.880287 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:26.880270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"]
Jul 02 16:58:27.018229 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16b7547d-4787-4c14-89ec-e781a2deee60-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.018229 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018233    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16b7547d-4787-4c14-89ec-e781a2deee60-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.018412 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018250    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22d0a80-bc7c-4979-8a72-5df601bcf20b-tmp\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.018412 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018333    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7fv\" (UniqueName: \"kubernetes.io/projected/e22d0a80-bc7c-4979-8a72-5df601bcf20b-kube-api-access-9j7fv\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.018412 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.018412 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018405    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e22d0a80-bc7c-4979-8a72-5df601bcf20b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.018535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.018535 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018515    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea676d64-9e98-460e-b414-9a46cda3afc8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.018596 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.018630 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.018630 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.018617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2px\" (UniqueName: \"kubernetes.io/projected/ea676d64-9e98-460e-b414-9a46cda3afc8-kube-api-access-gz2px\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.119937 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.119828    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e22d0a80-bc7c-4979-8a72-5df601bcf20b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.119937 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.119905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.119957    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea676d64-9e98-460e-b414-9a46cda3afc8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.119993    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2px\" (UniqueName: \"kubernetes.io/projected/ea676d64-9e98-460e-b414-9a46cda3afc8-kube-api-access-gz2px\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120092    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16b7547d-4787-4c14-89ec-e781a2deee60-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.120151 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120138    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16b7547d-4787-4c14-89ec-e781a2deee60-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.120446 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22d0a80-bc7c-4979-8a72-5df601bcf20b-tmp\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.120446 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120250    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7fv\" (UniqueName: \"kubernetes.io/projected/e22d0a80-bc7c-4979-8a72-5df601bcf20b-kube-api-access-9j7fv\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.120446 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120277    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.120991 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.120967    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea676d64-9e98-460e-b414-9a46cda3afc8-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.121303 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.121225    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22d0a80-bc7c-4979-8a72-5df601bcf20b-tmp\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.121303 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.121153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/16b7547d-4787-4c14-89ec-e781a2deee60-nginx-conf\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.123249 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.123206    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-ca\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.123473 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.123447    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/e22d0a80-bc7c-4979-8a72-5df601bcf20b-klusterlet-config\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.123569 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.123477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.123825 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.123803    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.124137 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.124114    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/16b7547d-4787-4c14-89ec-e781a2deee60-networking-console-plugin-cert\") pod \"networking-console-plugin-6cdb5968b9-c6vrg\" (UID: \"16b7547d-4787-4c14-89ec-e781a2deee60\") " pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.124728 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.124702    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/ea676d64-9e98-460e-b414-9a46cda3afc8-hub\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.131127 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.131079    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2px\" (UniqueName: \"kubernetes.io/projected/ea676d64-9e98-460e-b414-9a46cda3afc8-kube-api-access-gz2px\") pod \"cluster-proxy-proxy-agent-7dfff88445-dxj26\" (UID: \"ea676d64-9e98-460e-b414-9a46cda3afc8\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.141681 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.141652    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7fv\" (UniqueName: \"kubernetes.io/projected/e22d0a80-bc7c-4979-8a72-5df601bcf20b-kube-api-access-9j7fv\") pod \"klusterlet-addon-workmgr-b8bc77fcc-cv56b\" (UID: \"e22d0a80-bc7c-4979-8a72-5df601bcf20b\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.154345 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.154323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"
Jul 02 16:58:27.162449 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.162426    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:27.185227 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.184988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"
Jul 02 16:58:27.287429 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.280915    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-565sb"
Jul 02 16:58:27.390175 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.388774    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg"]
Jul 02 16:58:27.391017 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:27.390831    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16b7547d_4787_4c14_89ec_e781a2deee60.slice/crio-bbe31b448833e158f6ae5b89d20c7aa2143279bb446b58121a28a0dcf792797b WatchSource:0}: Error finding container bbe31b448833e158f6ae5b89d20c7aa2143279bb446b58121a28a0dcf792797b: Status 404 returned error can't find the container with id bbe31b448833e158f6ae5b89d20c7aa2143279bb446b58121a28a0dcf792797b
Jul 02 16:58:27.391920 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.391253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhb94" event={"ID":"b9a39120-edd8-43ac-b7ff-016eee8f0658","Type":"ContainerStarted","Data":"89c4ae93753364fc7ca3c264a0408ab5926caa83a491d2b2a6976f8412fbc12f"}
Jul 02 16:58:27.401692 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.401516    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"]
Jul 02 16:58:27.433434 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:27.433404    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26"]
Jul 02 16:58:27.438599 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:27.438565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea676d64_9e98_460e_b414_9a46cda3afc8.slice/crio-c366f95075dab589174021900c9769dc5b401960c2ef96b90dfb885619aac240 WatchSource:0}: Error finding container c366f95075dab589174021900c9769dc5b401960c2ef96b90dfb885619aac240: Status 404 returned error can't find the container with id c366f95075dab589174021900c9769dc5b401960c2ef96b90dfb885619aac240
Jul 02 16:58:28.397471 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:28.397386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b" event={"ID":"e22d0a80-bc7c-4979-8a72-5df601bcf20b","Type":"ContainerStarted","Data":"5cb35d673029530d9b684174a34b6b131e7afe4f48660b6e673a62cc8537cb21"}
Jul 02 16:58:28.402152 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:28.402070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg" event={"ID":"16b7547d-4787-4c14-89ec-e781a2deee60","Type":"ContainerStarted","Data":"bbe31b448833e158f6ae5b89d20c7aa2143279bb446b58121a28a0dcf792797b"}
Jul 02 16:58:28.403693 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:28.403633    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26" event={"ID":"ea676d64-9e98-460e-b414-9a46cda3afc8","Type":"ContainerStarted","Data":"c366f95075dab589174021900c9769dc5b401960c2ef96b90dfb885619aac240"}
Jul 02 16:58:29.039127 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.039090    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:58:29.042564 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.042537    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"image-registry-5dd8974cf-tlvxg\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") " pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:58:29.333431 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.333317    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lw9nb\""
Jul 02 16:58:29.340036 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.340006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:58:29.847476 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.847440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:58:29.851207 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.851154    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8-metrics-tls\") pod \"dns-default-q28mw\" (UID: \"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8\") " pod="openshift-dns/dns-default-q28mw"
Jul 02 16:58:29.950766 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.950735    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gz8mr\""
Jul 02 16:58:29.957789 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:29.957703    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-q28mw"
Jul 02 16:58:33.708659 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:33.708566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:58:33.715766 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:33.715686    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc77a624a_c496_44bf_a03f_10a1c5e38c7c.slice/crio-ee2941e9ade170ee6e307ca885c8ef0305fe0581f1a51725bc3e32c58b602968 WatchSource:0}: Error finding container ee2941e9ade170ee6e307ca885c8ef0305fe0581f1a51725bc3e32c58b602968: Status 404 returned error can't find the container with id ee2941e9ade170ee6e307ca885c8ef0305fe0581f1a51725bc3e32c58b602968
Jul 02 16:58:33.742914 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:33.742655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-q28mw"]
Jul 02 16:58:33.750765 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:33.750733    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbea20f0_9b8d_48a2_86ae_8a99d6c4dca8.slice/crio-3b4b61fa93c9d1f821e24fecaafb563b7fce28c960a4760048500f7976097208 WatchSource:0}: Error finding container 3b4b61fa93c9d1f821e24fecaafb563b7fce28c960a4760048500f7976097208: Status 404 returned error can't find the container with id 3b4b61fa93c9d1f821e24fecaafb563b7fce28c960a4760048500f7976097208
Jul 02 16:58:34.427281 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.427244    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhb94" event={"ID":"b9a39120-edd8-43ac-b7ff-016eee8f0658","Type":"ContainerStarted","Data":"b713a02ed794c3e1d9f0d5fcf236a3ac3bb7197eea27f8daddee452eca6a3a98"}
Jul 02 16:58:34.427546 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.427319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhb94" event={"ID":"b9a39120-edd8-43ac-b7ff-016eee8f0658","Type":"ContainerStarted","Data":"de0c85fb0503ee9f2242c340296867e1b7a39873efe92cf30a75cdb3f5788c46"}
Jul 02 16:58:34.428883 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.428838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b" event={"ID":"e22d0a80-bc7c-4979-8a72-5df601bcf20b","Type":"ContainerStarted","Data":"ea09cf22663a2b136a5ac51300dd96650eabffbf43201430e8817d4d71083263"}
Jul 02 16:58:34.429108 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.429060    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:34.430799 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.430777    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b"
Jul 02 16:58:34.430978 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.430955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg" event={"ID":"16b7547d-4787-4c14-89ec-e781a2deee60","Type":"ContainerStarted","Data":"d5887f06a44dcbd350a112ecc95fefc664ea3f7122925f8ddf7d42fe6d7da294"}
Jul 02 16:58:34.432576 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.432552    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" event={"ID":"c77a624a-c496-44bf-a03f-10a1c5e38c7c","Type":"ContainerStarted","Data":"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"}
Jul 02 16:58:34.432677 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.432594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" event={"ID":"c77a624a-c496-44bf-a03f-10a1c5e38c7c","Type":"ContainerStarted","Data":"ee2941e9ade170ee6e307ca885c8ef0305fe0581f1a51725bc3e32c58b602968"}
Jul 02 16:58:34.432677 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.432643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:58:34.433702 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.433683    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q28mw" event={"ID":"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8","Type":"ContainerStarted","Data":"3b4b61fa93c9d1f821e24fecaafb563b7fce28c960a4760048500f7976097208"}
Jul 02 16:58:34.480149 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.480097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" podStartSLOduration=104.480082203 podStartE2EDuration="1m44.480082203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:34.479352292 +0000 UTC m=+106.166353275" watchObservedRunningTime="2026-07-02 16:58:34.480082203 +0000 UTC m=+106.167083174"
Jul 02 16:58:34.480319 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.480293    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-b8bc77fcc-cv56b" podStartSLOduration=2.364697444 podStartE2EDuration="8.480286806s" totalImagesPullingTime="6.115589362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:34.450715788 +0000 UTC m=+106.137716763" watchObservedRunningTime="2026-07-02 16:58:34.480286806 +0000 UTC m=+106.167287779"
Jul 02 16:58:34.538670 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:34.538550    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cdb5968b9-c6vrg" podStartSLOduration=2.41086757 podStartE2EDuration="8.538530456s" totalImagesPullingTime="6.127662886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:34.537216996 +0000 UTC m=+106.224217970" watchObservedRunningTime="2026-07-02 16:58:34.538530456 +0000 UTC m=+106.225531428"
Jul 02 16:58:36.446560 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:36.446518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q28mw" event={"ID":"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8","Type":"ContainerStarted","Data":"55d10428c4c2fd8cba97741077c83adc3c4ca987f3d65a95a9ec99a1c1fae213"}
Jul 02 16:58:36.447057 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:36.446614    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-q28mw" event={"ID":"dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8","Type":"ContainerStarted","Data":"c2220124de06f97a1bff5a08eeafffb7da289e4993158887202ba5a0aa2642b7"}
Jul 02 16:58:36.489601 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:36.489540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-q28mw" podStartSLOduration=69.82072209 podStartE2EDuration="1m11.489518138s" totalImagesPullingTime="1.668796048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:36.488682886 +0000 UTC m=+108.175683897" watchObservedRunningTime="2026-07-02 16:58:36.489518138 +0000 UTC m=+108.176519110"
Jul 02 16:58:37.450690 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:37.450658    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-q28mw"
Jul 02 16:58:38.457362 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.457260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26" event={"ID":"ea676d64-9e98-460e-b414-9a46cda3afc8","Type":"ContainerStarted","Data":"ce0c6bc6b81ba188c9c4d3fd0d474ecc31c11f3beb3074381193126d02c89a5a"}
Jul 02 16:58:38.457362 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.457301    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26" event={"ID":"ea676d64-9e98-460e-b414-9a46cda3afc8","Type":"ContainerStarted","Data":"6c716b94b8736c334749965deaf7f75dbf117aa8433bf9bf91998d8d9f13fcc1"}
Jul 02 16:58:38.457362 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.457319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26" event={"ID":"ea676d64-9e98-460e-b414-9a46cda3afc8","Type":"ContainerStarted","Data":"4fe13a3414c00e8399adb98dcdb0a56696f483a4a916ef07c8b717524f507550"}
Jul 02 16:58:38.461911 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.461886    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dhb94" event={"ID":"b9a39120-edd8-43ac-b7ff-016eee8f0658","Type":"ContainerStarted","Data":"780fb6ba1824bfc1dc4c5a99b394b3f27714105cd03f24aeb3153b1a2e104d0b"}
Jul 02 16:58:38.570214 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.570161    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7dfff88445-dxj26" podStartSLOduration=1.8992160999999999 podStartE2EDuration="12.570144853s" totalImagesPullingTime="10.670928753s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:58:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:38.51840364 +0000 UTC m=+110.205404612" watchObservedRunningTime="2026-07-02 16:58:38.570144853 +0000 UTC m=+110.257145910"
Jul 02 16:58:38.570940 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:38.570913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dhb94" podStartSLOduration=33.321090791 podStartE2EDuration="44.570903562s" totalImagesPullingTime="11.249812771s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 16:57:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:38.568794921 +0000 UTC m=+110.255795892" watchObservedRunningTime="2026-07-02 16:58:38.570903562 +0000 UTC m=+110.257904533"
Jul 02 16:58:42.720389 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.720352    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kdp2z"]
Jul 02 16:58:42.723765 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.723745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744055 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744025    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-root\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744143 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-textfile\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744143 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmdrs\" (UniqueName: \"kubernetes.io/projected/5eb8b86e-a545-4cb6-acd2-d1d89424721c-kube-api-access-vmdrs\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744217 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-metrics-client-ca\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744217 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744175    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744217 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744306 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744238    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-wtmp\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744306 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744260    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.744366 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.744306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-sys\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.753509 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.753482    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 02 16:58:42.753614 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.753566    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 02 16:58:42.753760 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.753741    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 02 16:58:42.753830 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.753769    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 02 16:58:42.754922 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.754907    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 02 16:58:42.754995 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.754966    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 02 16:58:42.755436 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.755422    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2vvhn\""
Jul 02 16:58:42.845553 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-wtmp\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-sys\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-root\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-textfile\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845705    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmdrs\" (UniqueName: \"kubernetes.io/projected/5eb8b86e-a545-4cb6-acd2-d1d89424721c-kube-api-access-vmdrs\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845717    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-sys\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845729    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-wtmp\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.845741 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-metrics-client-ca\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846155 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845726    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/5eb8b86e-a545-4cb6-acd2-d1d89424721c-root\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846155 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.845795    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846155 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:42.845960    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 02 16:58:42.846155 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:58:42.846022    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls podName:5eb8b86e-a545-4cb6-acd2-d1d89424721c nodeName:}" failed. No retries permitted until 2026-07-02 16:58:43.346001492 +0000 UTC m=+115.033002446 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls") pod "node-exporter-kdp2z" (UID: "5eb8b86e-a545-4cb6-acd2-d1d89424721c") : secret "node-exporter-tls" not found
Jul 02 16:58:42.846155 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.846085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846405 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.846312    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-textfile\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846664 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.846644    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.846664 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.846656    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5eb8b86e-a545-4cb6-acd2-d1d89424721c-metrics-client-ca\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.848351 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.848333    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:42.884386 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:42.884356    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmdrs\" (UniqueName: \"kubernetes.io/projected/5eb8b86e-a545-4cb6-acd2-d1d89424721c-kube-api-access-vmdrs\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:43.350552 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:43.350515    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:43.352791 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:43.352771    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/5eb8b86e-a545-4cb6-acd2-d1d89424721c-node-exporter-tls\") pod \"node-exporter-kdp2z\" (UID: \"5eb8b86e-a545-4cb6-acd2-d1d89424721c\") " pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:43.632427 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:43.632349    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdp2z"
Jul 02 16:58:43.644010 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:43.643983    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb8b86e_a545_4cb6_acd2_d1d89424721c.slice/crio-ece1579597c88dbb44b8ab53fea6c82c5209217c184697e4b29ee953ebe2dbb0 WatchSource:0}: Error finding container ece1579597c88dbb44b8ab53fea6c82c5209217c184697e4b29ee953ebe2dbb0: Status 404 returned error can't find the container with id ece1579597c88dbb44b8ab53fea6c82c5209217c184697e4b29ee953ebe2dbb0
Jul 02 16:58:44.482065 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:44.482038    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdp2z" event={"ID":"5eb8b86e-a545-4cb6-acd2-d1d89424721c","Type":"ContainerStarted","Data":"ece1579597c88dbb44b8ab53fea6c82c5209217c184697e4b29ee953ebe2dbb0"}
Jul 02 16:58:45.486963 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:45.486927    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eb8b86e-a545-4cb6-acd2-d1d89424721c" containerID="4bd0212800e5917948ba954f26107fa603bee423cb6d8d535635bf5cc84571e8" exitCode=0
Jul 02 16:58:45.487338 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:45.486977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdp2z" event={"ID":"5eb8b86e-a545-4cb6-acd2-d1d89424721c","Type":"ContainerDied","Data":"4bd0212800e5917948ba954f26107fa603bee423cb6d8d535635bf5cc84571e8"}
Jul 02 16:58:46.492823 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:46.492784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdp2z" event={"ID":"5eb8b86e-a545-4cb6-acd2-d1d89424721c","Type":"ContainerStarted","Data":"99bf6cf5e80fbf38f3d3113eb37c942c60f6265edc98d06e9bcb4aa130d6e327"}
Jul 02 16:58:46.493209 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:46.492841    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdp2z" event={"ID":"5eb8b86e-a545-4cb6-acd2-d1d89424721c","Type":"ContainerStarted","Data":"2efec3aa1d2b86ca2cde018fc11def01a626bcdf5a907c96e22f5d2fc3683b2d"}
Jul 02 16:58:47.464700 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:47.464671    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-q28mw"
Jul 02 16:58:47.540869 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:47.540816    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kdp2z" podStartSLOduration=4.649933195 podStartE2EDuration="5.540798509s" totalImagesPullingTime="828.436843ms" totalInitContainerRuntime="62.428471ms" isStatefulPod=true podCreationTimestamp="2026-07-02 16:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:58:46.541112174 +0000 UTC m=+118.228113146" watchObservedRunningTime="2026-07-02 16:58:47.540798509 +0000 UTC m=+119.227799462"
Jul 02 16:58:49.049440 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:49.049406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:58:49.055242 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:49.055219    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:58:58.781175 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:58.781139    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:58:58.783432 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:58.783411    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f22f8ef6-4542-4886-aed4-c4a7e8dcad53-metrics-certs\") pod \"network-metrics-daemon-srp4c\" (UID: \"f22f8ef6-4542-4886-aed4-c4a7e8dcad53\") " pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:58:58.830792 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:58.830764    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xzcqc\""
Jul 02 16:58:58.838600 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:58.838583    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-srp4c"
Jul 02 16:58:58.980888 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:58.980842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-srp4c"]
Jul 02 16:58:58.983839 ip-10-0-139-28 kubenswrapper[2581]: W0702 16:58:58.983816    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22f8ef6_4542_4886_aed4_c4a7e8dcad53.slice/crio-cdb1753b9d26c196093a3e6474be9b0e96349765d9a8cefe44670f7dbbaf59af WatchSource:0}: Error finding container cdb1753b9d26c196093a3e6474be9b0e96349765d9a8cefe44670f7dbbaf59af: Status 404 returned error can't find the container with id cdb1753b9d26c196093a3e6474be9b0e96349765d9a8cefe44670f7dbbaf59af
Jul 02 16:58:59.538307 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:58:59.538264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srp4c" event={"ID":"f22f8ef6-4542-4886-aed4-c4a7e8dcad53","Type":"ContainerStarted","Data":"cdb1753b9d26c196093a3e6474be9b0e96349765d9a8cefe44670f7dbbaf59af"}
Jul 02 16:59:00.545216 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:00.545175    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srp4c" event={"ID":"f22f8ef6-4542-4886-aed4-c4a7e8dcad53","Type":"ContainerStarted","Data":"6ac3fec48ba4a19126f88152b2f0cbf890b04225781b69ef06ed4516a4e750f0"}
Jul 02 16:59:00.545620 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:00.545241    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-srp4c" event={"ID":"f22f8ef6-4542-4886-aed4-c4a7e8dcad53","Type":"ContainerStarted","Data":"0f9e7327dfcd961a885b31ac2941e22ce7d8e58311591d87b997e15ac41a9737"}
Jul 02 16:59:00.573797 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:00.573744    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-srp4c" podStartSLOduration=129.664159746 podStartE2EDuration="2m10.573728933s" totalImagesPullingTime="909.569187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 16:56:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 16:59:00.573582986 +0000 UTC m=+132.260583953" watchObservedRunningTime="2026-07-02 16:59:00.573728933 +0000 UTC m=+132.260729905"
Jul 02 16:59:09.578729 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:09.578695    2581 generic.go:356] "Generic (PLEG): container finished" podID="54d70273-357a-456a-86aa-8f945e48f656" containerID="ab583f14c8f8c32300b2c710b87c650a4b28aa838c4ebcd5a4375374c6bc7d4c" exitCode=0
Jul 02 16:59:09.579148 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:09.578769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-bj6rc" event={"ID":"54d70273-357a-456a-86aa-8f945e48f656","Type":"ContainerDied","Data":"ab583f14c8f8c32300b2c710b87c650a4b28aa838c4ebcd5a4375374c6bc7d4c"}
Jul 02 16:59:09.579148 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:09.579118    2581 scope.go:117] "RemoveContainer" containerID="ab583f14c8f8c32300b2c710b87c650a4b28aa838c4ebcd5a4375374c6bc7d4c"
Jul 02 16:59:10.584292 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:10.584255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5f67b55555-bj6rc" event={"ID":"54d70273-357a-456a-86aa-8f945e48f656","Type":"ContainerStarted","Data":"7fcd5dd17363521ea1e354dad7ade1a9182e3c66052c4af8e5d54efd4367b7fd"}
Jul 02 16:59:14.069588 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.069552    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" podUID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" containerName="registry" containerID="cri-o://00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b" gracePeriod=30
Jul 02 16:59:14.320165 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.320110    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:59:14.410315 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410275    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410315 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410366    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410413    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ppb9c\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410470    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410523 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410778 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410530    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates\") pod \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\" (UID: \"c77a624a-c496-44bf-a03f-10a1c5e38c7c\") "
Jul 02 16:59:14.410831 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.410791    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 16:59:14.411062 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.411038    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 16:59:14.413256 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.413220    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 16:59:14.413363 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.413271    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 16:59:14.413363 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.413297    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 16:59:14.413363 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.413314    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 16:59:14.413467 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.413381    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c" (OuterVolumeSpecName: "kube-api-access-ppb9c") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "kube-api-access-ppb9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 16:59:14.418852 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.418825    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "c77a624a-c496-44bf-a03f-10a1c5e38c7c" (UID: "c77a624a-c496-44bf-a03f-10a1c5e38c7c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 16:59:14.511818 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511773    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-trusted-ca\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.511818 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511813    2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-bound-sa-token\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511827    2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-image-registry-private-configuration\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511840    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppb9c\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-kube-api-access-ppb9c\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511853    2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/c77a624a-c496-44bf-a03f-10a1c5e38c7c-installation-pull-secrets\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511889    2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-tls\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511900    2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/c77a624a-c496-44bf-a03f-10a1c5e38c7c-ca-trust-extracted\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.512047 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.511911    2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/c77a624a-c496-44bf-a03f-10a1c5e38c7c-registry-certificates\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 16:59:14.599299 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.599201    2581 generic.go:356] "Generic (PLEG): container finished" podID="6f2f11c5-ab72-4155-aa31-5c32c32d89cc" containerID="2720df380537d7a08efe6553ec3df819660a91ae14c4d2ff95d1f4a72cbd7446" exitCode=0
Jul 02 16:59:14.599299 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.599281    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld" event={"ID":"6f2f11c5-ab72-4155-aa31-5c32c32d89cc","Type":"ContainerDied","Data":"2720df380537d7a08efe6553ec3df819660a91ae14c4d2ff95d1f4a72cbd7446"}
Jul 02 16:59:14.599659 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.599643    2581 scope.go:117] "RemoveContainer" containerID="2720df380537d7a08efe6553ec3df819660a91ae14c4d2ff95d1f4a72cbd7446"
Jul 02 16:59:14.600505 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.600485    2581 generic.go:356] "Generic (PLEG): container finished" podID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" containerID="00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b" exitCode=0
Jul 02 16:59:14.600595 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.600528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" event={"ID":"c77a624a-c496-44bf-a03f-10a1c5e38c7c","Type":"ContainerDied","Data":"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"}
Jul 02 16:59:14.600595 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.600538    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg"
Jul 02 16:59:14.600595 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.600559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5dd8974cf-tlvxg" event={"ID":"c77a624a-c496-44bf-a03f-10a1c5e38c7c","Type":"ContainerDied","Data":"ee2941e9ade170ee6e307ca885c8ef0305fe0581f1a51725bc3e32c58b602968"}
Jul 02 16:59:14.600595 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.600580    2581 scope.go:117] "RemoveContainer" containerID="00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"
Jul 02 16:59:14.611983 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.611820    2581 scope.go:117] "RemoveContainer" containerID="00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"
Jul 02 16:59:14.612627 ip-10-0-139-28 kubenswrapper[2581]: E0702 16:59:14.612607    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b\": container with ID starting with 00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b not found: ID does not exist" containerID="00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"
Jul 02 16:59:14.612684 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.612635    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b"} err="failed to get container status \"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b\": rpc error: code = NotFound desc = could not find container \"00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b\": container with ID starting with 00add6abae5a45189d387678d550aa80eae7b1feea32b6008244b41c98d34d9b not found: ID does not exist"
Jul 02 16:59:14.641440 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.641412    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:59:14.646195 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.646170    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5dd8974cf-tlvxg"]
Jul 02 16:59:14.814961 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:14.814926    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" path="/var/lib/kubelet/pods/c77a624a-c496-44bf-a03f-10a1c5e38c7c/volumes"
Jul 02 16:59:15.608388 ip-10-0-139-28 kubenswrapper[2581]: I0702 16:59:15.608349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d7b7dc95c-7f7ld" event={"ID":"6f2f11c5-ab72-4155-aa31-5c32c32d89cc","Type":"ContainerStarted","Data":"5c62595d26808fc26f8a9ebbad18321530cabb2a2e68af409c2a344d01746730"}
Jul 02 17:01:48.743667 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:01:48.743635    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:01:48.753475 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:01:48.753447    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 02 17:01:48.756382 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:01:48.756364    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:02:52.594879 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.594787    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g58vn"]
Jul 02 17:02:52.595304 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.595067    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" containerName="registry"
Jul 02 17:02:52.595304 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.595078    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" containerName="registry"
Jul 02 17:02:52.595304 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.595130    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c77a624a-c496-44bf-a03f-10a1c5e38c7c" containerName="registry"
Jul 02 17:02:52.597826 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.597809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.600821 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.600802    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 02 17:02:52.600934 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.600914    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 02 17:02:52.602182 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.602167    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-vj997\""
Jul 02 17:02:52.612424 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.612402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g58vn"]
Jul 02 17:02:52.634287 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.634256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.634287 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.634285    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xct5m\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-kube-api-access-xct5m\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.734795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.734758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.734795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.734797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xct5m\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-kube-api-access-xct5m\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.750258 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.750229    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.750749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.750728    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xct5m\" (UniqueName: \"kubernetes.io/projected/3da644aa-8d34-408a-a65d-37e947e5dbfb-kube-api-access-xct5m\") pod \"cert-manager-webhook-6486ffd997-g58vn\" (UID: \"3da644aa-8d34-408a-a65d-37e947e5dbfb\") " pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:52.906825 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:52.906729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:53.030800 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:53.030604    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-g58vn"]
Jul 02 17:02:53.033468 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:02:53.033445    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3da644aa_8d34_408a_a65d_37e947e5dbfb.slice/crio-ed9a73f832c2fa799e93b965dd78aab4de52fb0694c0e625637a1d7b851ecd69 WatchSource:0}: Error finding container ed9a73f832c2fa799e93b965dd78aab4de52fb0694c0e625637a1d7b851ecd69: Status 404 returned error can't find the container with id ed9a73f832c2fa799e93b965dd78aab4de52fb0694c0e625637a1d7b851ecd69
Jul 02 17:02:53.035851 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:53.035834    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 17:02:53.363758 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:53.363726    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn" event={"ID":"3da644aa-8d34-408a-a65d-37e947e5dbfb","Type":"ContainerStarted","Data":"ed9a73f832c2fa799e93b965dd78aab4de52fb0694c0e625637a1d7b851ecd69"}
Jul 02 17:02:56.378513 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:56.378473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn" event={"ID":"3da644aa-8d34-408a-a65d-37e947e5dbfb","Type":"ContainerStarted","Data":"b7225119f7382061fe9c68377eeb6b9dfb1f6ebc476a8455a8586ef8b0fd455a"}
Jul 02 17:02:56.378926 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:56.378625    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:02:56.399807 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:02:56.399763    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn" podStartSLOduration=1.5511445670000001 podStartE2EDuration="4.399749554s" totalImagesPullingTime="2.848604987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:02:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:02:56.398017658 +0000 UTC m=+368.085018630" watchObservedRunningTime="2026-07-02 17:02:56.399749554 +0000 UTC m=+368.086750526"
Jul 02 17:03:02.384525 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:02.384495    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-g58vn"
Jul 02 17:03:26.619984 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.619934    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"]
Jul 02 17:03:26.623365 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.623344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.626293 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.626268    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-2dzb5\""
Jul 02 17:03:26.626670 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.626649    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 02 17:03:26.626941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.626923    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 02 17:03:26.627302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.627284    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 02 17:03:26.627375 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.627313    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 02 17:03:26.645752 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.645724    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"]
Jul 02 17:03:26.682723 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.682691    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.682974 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.682734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-tmp\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.682974 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.682830    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-kube-api-access-2jwnk\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.682974 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.682909    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-manifests\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.682974 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.682939    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.783550 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.783506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-manifests\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.783550 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.783552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.783749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.783611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.783749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.783678    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-tmp\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.783749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.783726    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-kube-api-access-2jwnk\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.784109 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.784085    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-manifests\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.786103 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.786075    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-tmp\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.786220 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.786204    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.786281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.786262    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-webhook-cert\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.802108 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.802089    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwnk\" (UniqueName: \"kubernetes.io/projected/028ed0a1-722d-4ee3-840b-6ee93a2d2c7c-kube-api-access-2jwnk\") pod \"opendatahub-operator-controller-manager-7758cbfb47-kmr74\" (UID: \"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c\") " pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:26.933621 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:26.933526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:27.071209 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:27.071185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"]
Jul 02 17:03:27.073833 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:03:27.073805    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028ed0a1_722d_4ee3_840b_6ee93a2d2c7c.slice/crio-ab9b57ee2bf18c8db07e8999080002cb13d6ed6937fa72fc4ec609b1bb677067 WatchSource:0}: Error finding container ab9b57ee2bf18c8db07e8999080002cb13d6ed6937fa72fc4ec609b1bb677067: Status 404 returned error can't find the container with id ab9b57ee2bf18c8db07e8999080002cb13d6ed6937fa72fc4ec609b1bb677067
Jul 02 17:03:27.488117 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:27.488073    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74" event={"ID":"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c","Type":"ContainerStarted","Data":"ab9b57ee2bf18c8db07e8999080002cb13d6ed6937fa72fc4ec609b1bb677067"}
Jul 02 17:03:30.500915 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:30.500853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74" event={"ID":"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c","Type":"ContainerStarted","Data":"c04806d444b5572ad7ec82bc6335c5d35d8d0e463078a5c98859795738c21c6d"}
Jul 02 17:03:31.505559 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:31.505460    2581 generic.go:356] "Generic (PLEG): container finished" podID="028ed0a1-722d-4ee3-840b-6ee93a2d2c7c" containerID="c04806d444b5572ad7ec82bc6335c5d35d8d0e463078a5c98859795738c21c6d" exitCode=0
Jul 02 17:03:31.505559 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:31.505547    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74" event={"ID":"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c","Type":"ContainerDied","Data":"c04806d444b5572ad7ec82bc6335c5d35d8d0e463078a5c98859795738c21c6d"}
Jul 02 17:03:32.511401 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:32.511361    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74" event={"ID":"028ed0a1-722d-4ee3-840b-6ee93a2d2c7c","Type":"ContainerStarted","Data":"01161235f5fb0fc328b9341ebe5bdbb7345411c7641b260291d8e70c8072254e"}
Jul 02 17:03:32.511792 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:32.511458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:32.535941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:32.535675    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74" podStartSLOduration=2.13569843 podStartE2EDuration="6.535652642s" totalImagesPullingTime="3.379589484s" totalInitContainerRuntime="1.020364728s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:32.5332017 +0000 UTC m=+404.220202693" watchObservedRunningTime="2026-07-02 17:03:32.535652642 +0000 UTC m=+404.222653616"
Jul 02 17:03:43.517237 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:43.517199    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7758cbfb47-kmr74"
Jul 02 17:03:46.680202 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.680164    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"]
Jul 02 17:03:46.684141 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.684120    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.689839 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.689810    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 02 17:03:46.689987 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.689892    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 02 17:03:46.691078 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.691059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 02 17:03:46.691238 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.691121    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ptgn5\""
Jul 02 17:03:46.691474 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.691453    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 02 17:03:46.698543 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.698521    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"]
Jul 02 17:03:46.844490 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.844454    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31cb3480-3219-4995-96b9-c01ee4243153-tmp\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.844710 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.844509    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l9hs\" (UniqueName: \"kubernetes.io/projected/31cb3480-3219-4995-96b9-c01ee4243153-kube-api-access-5l9hs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.844710 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.844594    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cb3480-3219-4995-96b9-c01ee4243153-tls-certs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.945976 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.945883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l9hs\" (UniqueName: \"kubernetes.io/projected/31cb3480-3219-4995-96b9-c01ee4243153-kube-api-access-5l9hs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.945976 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.945955    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cb3480-3219-4995-96b9-c01ee4243153-tls-certs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.946162 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.946017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31cb3480-3219-4995-96b9-c01ee4243153-tmp\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.948418 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.948388    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/31cb3480-3219-4995-96b9-c01ee4243153-tmp\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.948618 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.948599    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31cb3480-3219-4995-96b9-c01ee4243153-tls-certs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.963618 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.963586    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l9hs\" (UniqueName: \"kubernetes.io/projected/31cb3480-3219-4995-96b9-c01ee4243153-kube-api-access-5l9hs\") pod \"kube-auth-proxy-54f48b874f-6v8wt\" (UID: \"31cb3480-3219-4995-96b9-c01ee4243153\") " pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:46.996592 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:46.996560    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"
Jul 02 17:03:47.141844 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:47.141819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt"]
Jul 02 17:03:47.144482 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:03:47.144453    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31cb3480_3219_4995_96b9_c01ee4243153.slice/crio-ef2b65f5dcdd02b2cba0392b733ac8acaebd5262d3d717013a070ef082deb7ac WatchSource:0}: Error finding container ef2b65f5dcdd02b2cba0392b733ac8acaebd5262d3d717013a070ef082deb7ac: Status 404 returned error can't find the container with id ef2b65f5dcdd02b2cba0392b733ac8acaebd5262d3d717013a070ef082deb7ac
Jul 02 17:03:47.567941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:47.567891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt" event={"ID":"31cb3480-3219-4995-96b9-c01ee4243153","Type":"ContainerStarted","Data":"ef2b65f5dcdd02b2cba0392b733ac8acaebd5262d3d717013a070ef082deb7ac"}
Jul 02 17:03:48.293177 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.293139    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:03:48.296302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.296286    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.299092 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.299072    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 02 17:03:48.299188 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.299169    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 02 17:03:48.299188 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.299176    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 02 17:03:48.299282 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.299205    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6wc4g\""
Jul 02 17:03:48.299438 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.299423    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 02 17:03:48.312694 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.312674    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:03:48.460351 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460534 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460363    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460534 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszhg\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460534 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460422    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460534 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460462    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460745 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460532    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.460745 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.460602    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.561796 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.561711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.561984 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.561816    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562115 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.561979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562115 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.562030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562115 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.562061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562115 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.562085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszhg\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562309 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.562116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.562701 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.562643    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.565142 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.565094    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.565729 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.565699    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.566332 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.566310    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.566401 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.566318    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.571314 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.571288    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszhg\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.572493 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.571685    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token\") pod \"istiod-openshift-gateway-7574459656-9q9qr\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.605702 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.605675    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:48.765011 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:48.764672    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:03:48.769100 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:03:48.769072    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08ecf52_9d5e_4038_86dd_431140e41f78.slice/crio-e89300a7ba88cc05f3206e283f85e851acaa87588773a9263a84a95f326429c7 WatchSource:0}: Error finding container e89300a7ba88cc05f3206e283f85e851acaa87588773a9263a84a95f326429c7: Status 404 returned error can't find the container with id e89300a7ba88cc05f3206e283f85e851acaa87588773a9263a84a95f326429c7
Jul 02 17:03:49.579245 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:49.579209    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" event={"ID":"d08ecf52-9d5e-4038-86dd-431140e41f78","Type":"ContainerStarted","Data":"e89300a7ba88cc05f3206e283f85e851acaa87588773a9263a84a95f326429c7"}
Jul 02 17:03:49.614984 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:49.614962    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 02 17:03:50.585239 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:50.585198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt" event={"ID":"31cb3480-3219-4995-96b9-c01ee4243153","Type":"ContainerStarted","Data":"c56b4880b89f50046a9c1035879337e8c2d1892bcd33f17115bb530767af12f5"}
Jul 02 17:03:50.602448 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:50.602397    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54f48b874f-6v8wt" podStartSLOduration=2.136951209 podStartE2EDuration="4.602382624s" totalImagesPullingTime="2.465431415s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:50.601480284 +0000 UTC m=+422.288481279" watchObservedRunningTime="2026-07-02 17:03:50.602382624 +0000 UTC m=+422.289383596"
Jul 02 17:03:51.535783 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:51.535740    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:03:51.535906 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:51.535823    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:03:52.596329 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:52.596276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" event={"ID":"d08ecf52-9d5e-4038-86dd-431140e41f78","Type":"ContainerStarted","Data":"3676b7c345fa3d3b2795ca826697469d2c8e34fbeb0336c28da607c91ed8d219"}
Jul 02 17:03:52.596798 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:52.596440    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:52.598677 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:52.598646    2581 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-9q9qr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 02 17:03:52.598784 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:52.598705    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 02 17:03:52.620580 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:52.620532    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" podStartSLOduration=1.857107367 podStartE2EDuration="4.620517234s" totalImagesPullingTime="2.763409867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:03:52.618695091 +0000 UTC m=+424.305696062" watchObservedRunningTime="2026-07-02 17:03:52.620517234 +0000 UTC m=+424.307518206"
Jul 02 17:03:53.601108 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:53.601078    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:03:59.249806 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.249771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"]
Jul 02 17:03:59.253170 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.253154    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"]
Jul 02 17:03:59.253311 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.253292    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.256083 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.256065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.257408 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.257391    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 02 17:03:59.257491 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.257393    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-7knh7\""
Jul 02 17:03:59.259063 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.259038    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 02 17:03:59.259199 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.259186    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-bpdpj\""
Jul 02 17:03:59.280878 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.280838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"]
Jul 02 17:03:59.282032 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.282009    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"]
Jul 02 17:03:59.353235 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.353206    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.353436 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.353245    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7krgh\" (UniqueName: \"kubernetes.io/projected/f4b64ec2-c0cc-4d03-ab92-9631c302918a-kube-api-access-7krgh\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.353436 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.353318    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzwwn\" (UniqueName: \"kubernetes.io/projected/f9793e15-bd45-43b3-ad56-6aa147a2bedc-kube-api-access-lzwwn\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.353436 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.353340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.454334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.454298    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.454526 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.454345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7krgh\" (UniqueName: \"kubernetes.io/projected/f4b64ec2-c0cc-4d03-ab92-9631c302918a-kube-api-access-7krgh\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.454526 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:03:59.454457    2581 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 02 17:03:59.454642 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:03:59.454539    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert podName:f4b64ec2-c0cc-4d03-ab92-9631c302918a nodeName:}" failed. No retries permitted until 2026-07-02 17:03:59.95451897 +0000 UTC m=+431.641519920 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert") pod "kserve-controller-manager-d6f4b65b8-c2mtt" (UID: "f4b64ec2-c0cc-4d03-ab92-9631c302918a") : secret "kserve-webhook-server-cert" not found
Jul 02 17:03:59.454642 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.454457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzwwn\" (UniqueName: \"kubernetes.io/projected/f9793e15-bd45-43b3-ad56-6aa147a2bedc-kube-api-access-lzwwn\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.454759 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.454634    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.454802 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:03:59.454771    2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 02 17:03:59.454874 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:03:59.454846    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert podName:f9793e15-bd45-43b3-ad56-6aa147a2bedc nodeName:}" failed. No retries permitted until 2026-07-02 17:03:59.954827417 +0000 UTC m=+431.641828371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert") pod "llmisvc-controller-manager-777c6bd7d5-th7ks" (UID: "f9793e15-bd45-43b3-ad56-6aa147a2bedc") : secret "llmisvc-webhook-server-cert" not found
Jul 02 17:03:59.486545 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.486517    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzwwn\" (UniqueName: \"kubernetes.io/projected/f9793e15-bd45-43b3-ad56-6aa147a2bedc-kube-api-access-lzwwn\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.486744 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.486720    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7krgh\" (UniqueName: \"kubernetes.io/projected/f4b64ec2-c0cc-4d03-ab92-9631c302918a-kube-api-access-7krgh\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.958757 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.958715    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:03:59.958936 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.958768    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.961252 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.961223    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f9793e15-bd45-43b3-ad56-6aa147a2bedc-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-th7ks\" (UID: \"f9793e15-bd45-43b3-ad56-6aa147a2bedc\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:03:59.961368 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:03:59.961329    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f4b64ec2-c0cc-4d03-ab92-9631c302918a-cert\") pod \"kserve-controller-manager-d6f4b65b8-c2mtt\" (UID: \"f4b64ec2-c0cc-4d03-ab92-9631c302918a\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:04:00.164700 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.164663    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:04:00.170139 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.170115    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:04:00.319161 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.318984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"]
Jul 02 17:04:00.322036 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:04:00.322007    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b64ec2_c0cc_4d03_ab92_9631c302918a.slice/crio-118209a1913d5838377c33d3398dc2e346d4bad2da77adb3b1d2ae85e00dfd5e WatchSource:0}: Error finding container 118209a1913d5838377c33d3398dc2e346d4bad2da77adb3b1d2ae85e00dfd5e: Status 404 returned error can't find the container with id 118209a1913d5838377c33d3398dc2e346d4bad2da77adb3b1d2ae85e00dfd5e
Jul 02 17:04:00.338452 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.338374    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"]
Jul 02 17:04:00.340712 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:04:00.340677    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf9793e15_bd45_43b3_ad56_6aa147a2bedc.slice/crio-323e260c03ae1f3a731d27363bea1ecadc21c5619651981899af26c8d4d49be9 WatchSource:0}: Error finding container 323e260c03ae1f3a731d27363bea1ecadc21c5619651981899af26c8d4d49be9: Status 404 returned error can't find the container with id 323e260c03ae1f3a731d27363bea1ecadc21c5619651981899af26c8d4d49be9
Jul 02 17:04:00.628368 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.628327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt" event={"ID":"f4b64ec2-c0cc-4d03-ab92-9631c302918a","Type":"ContainerStarted","Data":"118209a1913d5838377c33d3398dc2e346d4bad2da77adb3b1d2ae85e00dfd5e"}
Jul 02 17:04:00.629374 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:00.629349    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks" event={"ID":"f9793e15-bd45-43b3-ad56-6aa147a2bedc","Type":"ContainerStarted","Data":"323e260c03ae1f3a731d27363bea1ecadc21c5619651981899af26c8d4d49be9"}
Jul 02 17:04:03.482900 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.482875    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:04:03.483222 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.483065    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerName="discovery" containerID="cri-o://3676b7c345fa3d3b2795ca826697469d2c8e34fbeb0336c28da607c91ed8d219" gracePeriod=30
Jul 02 17:04:03.646934 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.646894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt" event={"ID":"f4b64ec2-c0cc-4d03-ab92-9631c302918a","Type":"ContainerStarted","Data":"9ff067a4c0722e4368c0587c8cb3e98c4aa00264622f6833e508070555631517"}
Jul 02 17:04:03.647123 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.647002    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:04:03.648646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.648618    2581 generic.go:356] "Generic (PLEG): container finished" podID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerID="3676b7c345fa3d3b2795ca826697469d2c8e34fbeb0336c28da607c91ed8d219" exitCode=0
Jul 02 17:04:03.648752 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.648696    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" event={"ID":"d08ecf52-9d5e-4038-86dd-431140e41f78","Type":"ContainerDied","Data":"3676b7c345fa3d3b2795ca826697469d2c8e34fbeb0336c28da607c91ed8d219"}
Jul 02 17:04:03.650227 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.650205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks" event={"ID":"f9793e15-bd45-43b3-ad56-6aa147a2bedc","Type":"ContainerStarted","Data":"b4f80af0b5f481158c995a1524ddf70443461ce84cda1a98c7d7292681869a69"}
Jul 02 17:04:03.650293 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.650280    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:04:03.669499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.669454    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt" podStartSLOduration=1.535400893 podStartE2EDuration="4.669441778s" totalImagesPullingTime="3.134040885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:03.667141118 +0000 UTC m=+435.354142090" watchObservedRunningTime="2026-07-02 17:04:03.669441778 +0000 UTC m=+435.356442757"
Jul 02 17:04:03.694174 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.694115    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks" podStartSLOduration=1.5807292469999998 podStartE2EDuration="4.694094115s" totalImagesPullingTime="3.113364868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:03.691607567 +0000 UTC m=+435.378608562" watchObservedRunningTime="2026-07-02 17:04:03.694094115 +0000 UTC m=+435.381095088"
Jul 02 17:04:03.734888 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.734847    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:04:03.892776 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.892743    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.892986 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.892812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.892986 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.892844    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.893107 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.892985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.893107 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.893027    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.893107 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.893053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.893107 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.893091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qszhg\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg\") pod \"d08ecf52-9d5e-4038-86dd-431140e41f78\" (UID: \"d08ecf52-9d5e-4038-86dd-431140e41f78\") "
Jul 02 17:04:03.893387 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.893352    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 17:04:03.895487 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895456    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs" (OuterVolumeSpecName: "local-certs") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:04:03.895594 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895575    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token" (OuterVolumeSpecName: "istio-token") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:04:03.895637 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895569    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:04:03.895637 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895605    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg" (OuterVolumeSpecName: "kube-api-access-qszhg") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "kube-api-access-qszhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:04:03.895637 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895620    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts" (OuterVolumeSpecName: "cacerts") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:04:03.895748 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.895683    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "d08ecf52-9d5e-4038-86dd-431140e41f78" (UID: "d08ecf52-9d5e-4038-86dd-431140e41f78"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994245    2581 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-cacerts\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994277    2581 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-kubeconfig\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994289    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-ca-configmap\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994299    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-token\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994308    2581 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d08ecf52-9d5e-4038-86dd-431140e41f78-istio-csr-dns-cert\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994316    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszhg\" (UniqueName: \"kubernetes.io/projected/d08ecf52-9d5e-4038-86dd-431140e41f78-kube-api-access-qszhg\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:03.994334 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:03.994324    2581 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d08ecf52-9d5e-4038-86dd-431140e41f78-local-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:04.662586 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.662555    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"
Jul 02 17:04:04.662586 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.662585    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr" event={"ID":"d08ecf52-9d5e-4038-86dd-431140e41f78","Type":"ContainerDied","Data":"e89300a7ba88cc05f3206e283f85e851acaa87588773a9263a84a95f326429c7"}
Jul 02 17:04:04.663092 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.662647    2581 scope.go:117] "RemoveContainer" containerID="3676b7c345fa3d3b2795ca826697469d2c8e34fbeb0336c28da607c91ed8d219"
Jul 02 17:04:04.691106 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.691078    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:04:04.694745 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.694722    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-9q9qr"]
Jul 02 17:04:04.814646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:04.814614    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" path="/var/lib/kubelet/pods/d08ecf52-9d5e-4038-86dd-431140e41f78/volumes"
Jul 02 17:04:34.665128 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:34.665093    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-th7ks"
Jul 02 17:04:34.668053 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:34.668030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-c2mtt"
Jul 02 17:04:47.809604 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.809565    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:47.810198 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.810025    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerName="discovery"
Jul 02 17:04:47.810198 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.810046    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerName="discovery"
Jul 02 17:04:47.810198 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.810124    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d08ecf52-9d5e-4038-86dd-431140e41f78" containerName="discovery"
Jul 02 17:04:47.813211 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.813185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:47.815881 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.815840    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 02 17:04:47.817028 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.817010    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 02 17:04:47.817145 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.817095    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kvzfk\""
Jul 02 17:04:47.822965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.822936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:47.930747 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:47.930715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqv7r\" (UniqueName: \"kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r\") pod \"kuadrant-operator-catalog-k2fpt\" (UID: \"834e5845-bf0e-4aee-a1a2-962d01bb93a4\") " pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:48.032170 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.032130    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqv7r\" (UniqueName: \"kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r\") pod \"kuadrant-operator-catalog-k2fpt\" (UID: \"834e5845-bf0e-4aee-a1a2-962d01bb93a4\") " pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:48.043441 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.043415    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqv7r\" (UniqueName: \"kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r\") pod \"kuadrant-operator-catalog-k2fpt\" (UID: \"834e5845-bf0e-4aee-a1a2-962d01bb93a4\") " pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:48.124832 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.124745    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:48.179170 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.179133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:48.258217 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.258192    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:48.260715 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:04:48.260692    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod834e5845_bf0e_4aee_a1a2_962d01bb93a4.slice/crio-e14c9cbc1602919a3e8101b0a9e467c53566acdd0c21dae3463ac383ca2810ba WatchSource:0}: Error finding container e14c9cbc1602919a3e8101b0a9e467c53566acdd0c21dae3463ac383ca2810ba: Status 404 returned error can't find the container with id e14c9cbc1602919a3e8101b0a9e467c53566acdd0c21dae3463ac383ca2810ba
Jul 02 17:04:48.388582 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.388508    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhtj7"]
Jul 02 17:04:48.392614 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.392596    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:48.400915 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.400890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhtj7"]
Jul 02 17:04:48.537132 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.537094    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnm7d\" (UniqueName: \"kubernetes.io/projected/9e8a5974-2b52-4180-b1fd-a0f5f3d055d8-kube-api-access-lnm7d\") pod \"kuadrant-operator-catalog-zhtj7\" (UID: \"9e8a5974-2b52-4180-b1fd-a0f5f3d055d8\") " pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:48.638507 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.638475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnm7d\" (UniqueName: \"kubernetes.io/projected/9e8a5974-2b52-4180-b1fd-a0f5f3d055d8-kube-api-access-lnm7d\") pod \"kuadrant-operator-catalog-zhtj7\" (UID: \"9e8a5974-2b52-4180-b1fd-a0f5f3d055d8\") " pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:48.647092 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.647030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnm7d\" (UniqueName: \"kubernetes.io/projected/9e8a5974-2b52-4180-b1fd-a0f5f3d055d8-kube-api-access-lnm7d\") pod \"kuadrant-operator-catalog-zhtj7\" (UID: \"9e8a5974-2b52-4180-b1fd-a0f5f3d055d8\") " pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:48.702163 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.702129    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:48.846070 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.846034    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" event={"ID":"834e5845-bf0e-4aee-a1a2-962d01bb93a4","Type":"ContainerStarted","Data":"e14c9cbc1602919a3e8101b0a9e467c53566acdd0c21dae3463ac383ca2810ba"}
Jul 02 17:04:48.856527 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:48.856501    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhtj7"]
Jul 02 17:04:48.858897 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:04:48.858870    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8a5974_2b52_4180_b1fd_a0f5f3d055d8.slice/crio-c977960e3cc2163367062ffe3423b5a85d9e2fb14585d518452cc351ba9329a8 WatchSource:0}: Error finding container c977960e3cc2163367062ffe3423b5a85d9e2fb14585d518452cc351ba9329a8: Status 404 returned error can't find the container with id c977960e3cc2163367062ffe3423b5a85d9e2fb14585d518452cc351ba9329a8
Jul 02 17:04:49.852070 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:49.852035    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7" event={"ID":"9e8a5974-2b52-4180-b1fd-a0f5f3d055d8","Type":"ContainerStarted","Data":"c977960e3cc2163367062ffe3423b5a85d9e2fb14585d518452cc351ba9329a8"}
Jul 02 17:04:50.857026 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:50.856987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" event={"ID":"834e5845-bf0e-4aee-a1a2-962d01bb93a4","Type":"ContainerStarted","Data":"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"}
Jul 02 17:04:50.857466 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:50.857011    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" podUID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" containerName="registry-server" containerID="cri-o://70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296" gracePeriod=2
Jul 02 17:04:50.858486 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:50.858464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7" event={"ID":"9e8a5974-2b52-4180-b1fd-a0f5f3d055d8","Type":"ContainerStarted","Data":"42a7b862c916e882c52028cd340dd8a327c14aa5ad0156c0fa2b731995e7a6f7"}
Jul 02 17:04:50.873483 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:50.873440    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" podStartSLOduration=1.8846563 podStartE2EDuration="3.87342671s" totalImagesPullingTime="1.98877041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:04:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:50.871722327 +0000 UTC m=+482.558723300" watchObservedRunningTime="2026-07-02 17:04:50.87342671 +0000 UTC m=+482.560427730"
Jul 02 17:04:50.887907 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:50.887845    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7" podStartSLOduration=1.496268012 podStartE2EDuration="2.887829248s" totalImagesPullingTime="1.391561236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:04:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:04:50.887604307 +0000 UTC m=+482.574605280" watchObservedRunningTime="2026-07-02 17:04:50.887829248 +0000 UTC m=+482.574830240"
Jul 02 17:04:51.097492 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.097462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:51.159903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.159797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqv7r\" (UniqueName: \"kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r\") pod \"834e5845-bf0e-4aee-a1a2-962d01bb93a4\" (UID: \"834e5845-bf0e-4aee-a1a2-962d01bb93a4\") "
Jul 02 17:04:51.162056 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.162031    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r" (OuterVolumeSpecName: "kube-api-access-fqv7r") pod "834e5845-bf0e-4aee-a1a2-962d01bb93a4" (UID: "834e5845-bf0e-4aee-a1a2-962d01bb93a4"). InnerVolumeSpecName "kube-api-access-fqv7r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:04:51.260667 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.260618    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqv7r\" (UniqueName: \"kubernetes.io/projected/834e5845-bf0e-4aee-a1a2-962d01bb93a4-kube-api-access-fqv7r\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:04:51.863309 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.863273    2581 generic.go:356] "Generic (PLEG): container finished" podID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" containerID="70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296" exitCode=0
Jul 02 17:04:51.863777 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.863319    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt"
Jul 02 17:04:51.863777 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.863354    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" event={"ID":"834e5845-bf0e-4aee-a1a2-962d01bb93a4","Type":"ContainerDied","Data":"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"}
Jul 02 17:04:51.863777 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.863405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-k2fpt" event={"ID":"834e5845-bf0e-4aee-a1a2-962d01bb93a4","Type":"ContainerDied","Data":"e14c9cbc1602919a3e8101b0a9e467c53566acdd0c21dae3463ac383ca2810ba"}
Jul 02 17:04:51.863777 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.863426    2581 scope.go:117] "RemoveContainer" containerID="70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"
Jul 02 17:04:51.872335 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.872320    2581 scope.go:117] "RemoveContainer" containerID="70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"
Jul 02 17:04:51.872592 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:04:51.872575    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296\": container with ID starting with 70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296 not found: ID does not exist" containerID="70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"
Jul 02 17:04:51.872637 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.872603    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296"} err="failed to get container status \"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296\": rpc error: code = NotFound desc = could not find container \"70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296\": container with ID starting with 70f60d2371570ab618d48210c7eacb09b2eb9b323dcd3f9a1a1fc10365b98296 not found: ID does not exist"
Jul 02 17:04:51.889650 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.889621    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:51.898543 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:51.898520    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-k2fpt"]
Jul 02 17:04:52.815590 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:52.815558    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" path="/var/lib/kubelet/pods/834e5845-bf0e-4aee-a1a2-962d01bb93a4/volumes"
Jul 02 17:04:58.703117 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:58.703077    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:58.703540 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:58.703143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:58.725910 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:58.725884    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:04:58.916394 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:04:58.916351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-zhtj7"
Jul 02 17:05:14.379489 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.379449    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"]
Jul 02 17:05:14.380023 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.379893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" containerName="registry-server"
Jul 02 17:05:14.380023 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.379912    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" containerName="registry-server"
Jul 02 17:05:14.380023 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.380021    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="834e5845-bf0e-4aee-a1a2-962d01bb93a4" containerName="registry-server"
Jul 02 17:05:14.387802 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.387775    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:14.394080 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.394033    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6fq62\""
Jul 02 17:05:14.396091 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.396065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"]
Jul 02 17:05:14.551195 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.551151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd5x2\" (UniqueName: \"kubernetes.io/projected/29cd7b26-ce52-4976-bf50-45812e418bd4-kube-api-access-kd5x2\") pod \"authorino-operator-75cc8d5cb7-5rzqk\" (UID: \"29cd7b26-ce52-4976-bf50-45812e418bd4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:14.651687 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.651579    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd5x2\" (UniqueName: \"kubernetes.io/projected/29cd7b26-ce52-4976-bf50-45812e418bd4-kube-api-access-kd5x2\") pod \"authorino-operator-75cc8d5cb7-5rzqk\" (UID: \"29cd7b26-ce52-4976-bf50-45812e418bd4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:14.666556 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.666519    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd5x2\" (UniqueName: \"kubernetes.io/projected/29cd7b26-ce52-4976-bf50-45812e418bd4-kube-api-access-kd5x2\") pod \"authorino-operator-75cc8d5cb7-5rzqk\" (UID: \"29cd7b26-ce52-4976-bf50-45812e418bd4\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:14.698328 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.698291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:14.840204 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.840180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"]
Jul 02 17:05:14.842294 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:14.842271    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29cd7b26_ce52_4976_bf50_45812e418bd4.slice/crio-7e51dcbb1a51dcc0006003a26ff36cda805d43e37dd4af3898f6bd334e8d48f7 WatchSource:0}: Error finding container 7e51dcbb1a51dcc0006003a26ff36cda805d43e37dd4af3898f6bd334e8d48f7: Status 404 returned error can't find the container with id 7e51dcbb1a51dcc0006003a26ff36cda805d43e37dd4af3898f6bd334e8d48f7
Jul 02 17:05:14.958331 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:14.958251    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk" event={"ID":"29cd7b26-ce52-4976-bf50-45812e418bd4","Type":"ContainerStarted","Data":"7e51dcbb1a51dcc0006003a26ff36cda805d43e37dd4af3898f6bd334e8d48f7"}
Jul 02 17:05:16.969529 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:16.969487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk" event={"ID":"29cd7b26-ce52-4976-bf50-45812e418bd4","Type":"ContainerStarted","Data":"ffff13868b2e8e027c78d75343ae3a6f3222706b3e97ed8c9570ef9f1bceb270"}
Jul 02 17:05:16.969979 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:16.969629    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:16.987852 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:16.987808    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk" podStartSLOduration=1.008871266 podStartE2EDuration="2.987792489s" totalImagesPullingTime="1.978921223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:16.98748262 +0000 UTC m=+508.674483676" watchObservedRunningTime="2026-07-02 17:05:16.987792489 +0000 UTC m=+508.674793463"
Jul 02 17:05:17.951949 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:17.951912    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"]
Jul 02 17:05:17.955144 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:17.955127    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:17.958638 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:17.958608    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p8crq\""
Jul 02 17:05:17.969268 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:17.969242    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"]
Jul 02 17:05:18.080325 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.080289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.080688 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.080337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mcjd\" (UniqueName: \"kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.181314 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.181274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.181314 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.181323    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mcjd\" (UniqueName: \"kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.181653 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.181633    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.195763 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.195733    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mcjd\" (UniqueName: \"kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd\") pod \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.265575 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.265485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:18.413887 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.413846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"]
Jul 02 17:05:18.415759 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:18.415734    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54a2614_7259_4a7b_9fd1_908b5f3a34ba.slice/crio-9c39e297ec72e984937916fd07b1994191c2b33276a3ef274ac3474d47d0b0a3 WatchSource:0}: Error finding container 9c39e297ec72e984937916fd07b1994191c2b33276a3ef274ac3474d47d0b0a3: Status 404 returned error can't find the container with id 9c39e297ec72e984937916fd07b1994191c2b33276a3ef274ac3474d47d0b0a3
Jul 02 17:05:18.979922 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:18.979889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt" event={"ID":"e54a2614-7259-4a7b-9fd1-908b5f3a34ba","Type":"ContainerStarted","Data":"9c39e297ec72e984937916fd07b1994191c2b33276a3ef274ac3474d47d0b0a3"}
Jul 02 17:05:20.437737 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.437703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"]
Jul 02 17:05:20.442180 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.442156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:20.446319 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.446296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jnllh\""
Jul 02 17:05:20.463903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.463874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"]
Jul 02 17:05:20.494612 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.494533    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28h7c\" (UniqueName: \"kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c\") pod \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" (UID: \"bf688489-a09b-44b0-8f9c-ec0b01f8ff04\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:20.595175 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.595137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28h7c\" (UniqueName: \"kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c\") pod \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" (UID: \"bf688489-a09b-44b0-8f9c-ec0b01f8ff04\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:20.615287 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.615254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28h7c\" (UniqueName: \"kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c\") pod \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" (UID: \"bf688489-a09b-44b0-8f9c-ec0b01f8ff04\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:20.754722 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.754631    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:20.940953 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.940914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"]
Jul 02 17:05:20.944112 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:20.944084    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf688489_a09b_44b0_8f9c_ec0b01f8ff04.slice/crio-9bdf7b88a3c14f658f7842fc4349db1e2b991021942887297753fabf14192933 WatchSource:0}: Error finding container 9bdf7b88a3c14f658f7842fc4349db1e2b991021942887297753fabf14192933: Status 404 returned error can't find the container with id 9bdf7b88a3c14f658f7842fc4349db1e2b991021942887297753fabf14192933
Jul 02 17:05:20.992461 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:20.992415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm" event={"ID":"bf688489-a09b-44b0-8f9c-ec0b01f8ff04","Type":"ContainerStarted","Data":"9bdf7b88a3c14f658f7842fc4349db1e2b991021942887297753fabf14192933"}
Jul 02 17:05:22.524895 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.524840    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"]
Jul 02 17:05:22.528633 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.528602    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:22.533138 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.533117    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-dfhdf\""
Jul 02 17:05:22.533272 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.533169    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 02 17:05:22.542510 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.542488    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"]
Jul 02 17:05:22.613779 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.613743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlrq\" (UniqueName: \"kubernetes.io/projected/79208780-cdda-401c-8174-f1202f030f74-kube-api-access-wnlrq\") pod \"dns-operator-controller-manager-864dd78c4c-5lh94\" (UID: \"79208780-cdda-401c-8174-f1202f030f74\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:22.714813 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.714681    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlrq\" (UniqueName: \"kubernetes.io/projected/79208780-cdda-401c-8174-f1202f030f74-kube-api-access-wnlrq\") pod \"dns-operator-controller-manager-864dd78c4c-5lh94\" (UID: \"79208780-cdda-401c-8174-f1202f030f74\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:22.728405 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.728367    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlrq\" (UniqueName: \"kubernetes.io/projected/79208780-cdda-401c-8174-f1202f030f74-kube-api-access-wnlrq\") pod \"dns-operator-controller-manager-864dd78c4c-5lh94\" (UID: \"79208780-cdda-401c-8174-f1202f030f74\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:22.841700 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:22.841656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:23.373831 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:23.373807    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"]
Jul 02 17:05:23.376344 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:23.376319    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79208780_cdda_401c_8174_f1202f030f74.slice/crio-2a85daa7b3f8855a3d835b5d3766c6e7f71fd7139e728c8c82adaae54922a9d9 WatchSource:0}: Error finding container 2a85daa7b3f8855a3d835b5d3766c6e7f71fd7139e728c8c82adaae54922a9d9: Status 404 returned error can't find the container with id 2a85daa7b3f8855a3d835b5d3766c6e7f71fd7139e728c8c82adaae54922a9d9
Jul 02 17:05:24.013001 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.012967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt" event={"ID":"e54a2614-7259-4a7b-9fd1-908b5f3a34ba","Type":"ContainerStarted","Data":"7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb"}
Jul 02 17:05:24.013432 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.013100    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:24.014421 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.014393    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94" event={"ID":"79208780-cdda-401c-8174-f1202f030f74","Type":"ContainerStarted","Data":"2a85daa7b3f8855a3d835b5d3766c6e7f71fd7139e728c8c82adaae54922a9d9"}
Jul 02 17:05:24.015914 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.015894    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm" event={"ID":"bf688489-a09b-44b0-8f9c-ec0b01f8ff04","Type":"ContainerStarted","Data":"a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc"}
Jul 02 17:05:24.016052 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.016035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:24.052165 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.052117    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt" podStartSLOduration=2.243345668 podStartE2EDuration="7.052099581s" totalImagesPullingTime="4.808753913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:24.050319751 +0000 UTC m=+515.737320741" watchObservedRunningTime="2026-07-02 17:05:24.052099581 +0000 UTC m=+515.739100575"
Jul 02 17:05:24.093335 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:24.093274    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm" podStartSLOduration=1.7703465010000001 podStartE2EDuration="4.093255842s" totalImagesPullingTime="2.322909341s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:24.090373085 +0000 UTC m=+515.777374082" watchObservedRunningTime="2026-07-02 17:05:24.093255842 +0000 UTC m=+515.780256856"
Jul 02 17:05:26.025901 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.025839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94" event={"ID":"79208780-cdda-401c-8174-f1202f030f74","Type":"ContainerStarted","Data":"61b76ed83d9aa715612f9d3a24cc07bc63a6c584da72de8f943a2e2c07badb81"}
Jul 02 17:05:26.026283 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.025972    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:26.072052 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.071991    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94" podStartSLOduration=2.440238959 podStartE2EDuration="4.071977583s" totalImagesPullingTime="1.631738624s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:26.070811186 +0000 UTC m=+517.757812168" watchObservedRunningTime="2026-07-02 17:05:26.071977583 +0000 UTC m=+517.758978612"
Jul 02 17:05:26.725594 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.725555    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"]
Jul 02 17:05:26.729022 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.729006    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.732572 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.732550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-v6mf2\""
Jul 02 17:05:26.732771 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.732758    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 02 17:05:26.733010 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.732993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 02 17:05:26.749395 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.749367    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p9cg\" (UniqueName: \"kubernetes.io/projected/1203da23-9b52-49f7-a0d1-e227e9655f67-kube-api-access-5p9cg\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.749535 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.749434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1203da23-9b52-49f7-a0d1-e227e9655f67-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.749535 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.749512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1203da23-9b52-49f7-a0d1-e227e9655f67-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.751141 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.751116    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"]
Jul 02 17:05:26.850147 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.850113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p9cg\" (UniqueName: \"kubernetes.io/projected/1203da23-9b52-49f7-a0d1-e227e9655f67-kube-api-access-5p9cg\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.850315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.850161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1203da23-9b52-49f7-a0d1-e227e9655f67-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.850315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.850214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1203da23-9b52-49f7-a0d1-e227e9655f67-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.850843 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.850822    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1203da23-9b52-49f7-a0d1-e227e9655f67-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.852717 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.852694    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1203da23-9b52-49f7-a0d1-e227e9655f67-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:26.860033 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:26.860008    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p9cg\" (UniqueName: \"kubernetes.io/projected/1203da23-9b52-49f7-a0d1-e227e9655f67-kube-api-access-5p9cg\") pod \"kuadrant-console-plugin-66b5774985-25zsj\" (UID: \"1203da23-9b52-49f7-a0d1-e227e9655f67\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:27.038286 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:27.038209    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"
Jul 02 17:05:27.183000 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:27.182970    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj"]
Jul 02 17:05:27.186124 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:27.186084    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1203da23_9b52_49f7_a0d1_e227e9655f67.slice/crio-dd756ba3c9b9eeae881f0550d54a4d17aac3d6bb168c96f093df7b1585c245a2 WatchSource:0}: Error finding container dd756ba3c9b9eeae881f0550d54a4d17aac3d6bb168c96f093df7b1585c245a2: Status 404 returned error can't find the container with id dd756ba3c9b9eeae881f0550d54a4d17aac3d6bb168c96f093df7b1585c245a2
Jul 02 17:05:27.976999 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:27.976971    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-5rzqk"
Jul 02 17:05:28.037060 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:28.037023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj" event={"ID":"1203da23-9b52-49f7-a0d1-e227e9655f67","Type":"ContainerStarted","Data":"dd756ba3c9b9eeae881f0550d54a4d17aac3d6bb168c96f093df7b1585c245a2"}
Jul 02 17:05:35.022846 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:35.022802    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:35.023331 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:35.022923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:37.032150 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.032119    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5lh94"
Jul 02 17:05:37.294501 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.294468    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"]
Jul 02 17:05:37.294755 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.294725    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" containerName="manager" containerID="cri-o://7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb" gracePeriod=2
Jul 02 17:05:37.311676 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.311645    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"]
Jul 02 17:05:37.314415 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:37.314375    2581 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:37.330350 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.330314    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:37.330837 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.330771    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" containerName="manager"
Jul 02 17:05:37.330837 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.330790    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" containerName="manager"
Jul 02 17:05:37.330999 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.330907    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" containerName="manager"
Jul 02 17:05:37.345417 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.338022    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"]
Jul 02 17:05:37.345417 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.338070    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"]
Jul 02 17:05:37.345417 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.338238    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" containerName="manager" containerID="cri-o://a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc" gracePeriod=2
Jul 02 17:05:37.345417 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.338442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.348887 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:37.346242    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:37.371411 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.368448    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:37.374958 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.374896    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"]
Jul 02 17:05:37.375386 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.375363    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" containerName="manager"
Jul 02 17:05:37.375478 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.375387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" containerName="manager"
Jul 02 17:05:37.375478 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.375472    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" containerName="manager"
Jul 02 17:05:37.378849 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.378831    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:37.392882 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:37.392824    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:37.395552 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:37.395392    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:37.398208 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.398181    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"]
Jul 02 17:05:37.441694 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:37.441657    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:37.446060 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.446033    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5h6\" (UniqueName: \"kubernetes.io/projected/dccf5c59-6b9e-4b6b-a99c-73abf094447a-kube-api-access-pk5h6\") pod \"limitador-operator-controller-manager-859d7b78bd-nx9v6\" (UID: \"dccf5c59-6b9e-4b6b-a99c-73abf094447a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:37.446189 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.446081    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjk5j\" (UniqueName: \"kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.446189 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.446123    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.547549 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.547464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5h6\" (UniqueName: \"kubernetes.io/projected/dccf5c59-6b9e-4b6b-a99c-73abf094447a-kube-api-access-pk5h6\") pod \"limitador-operator-controller-manager-859d7b78bd-nx9v6\" (UID: \"dccf5c59-6b9e-4b6b-a99c-73abf094447a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:37.547549 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.547516    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjk5j\" (UniqueName: \"kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.547751 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.547551    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.548008 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.547989    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.566117 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.566080    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjk5j\" (UniqueName: \"kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7fksw\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.566244 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.566194    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5h6\" (UniqueName: \"kubernetes.io/projected/dccf5c59-6b9e-4b6b-a99c-73abf094447a-kube-api-access-pk5h6\") pod \"limitador-operator-controller-manager-859d7b78bd-nx9v6\" (UID: \"dccf5c59-6b9e-4b6b-a99c-73abf094447a\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:37.732654 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.732603    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:37.741183 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:37.741158    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:38.816341 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:38.816309    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:38.819393 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:38.819359    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.000111 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.000088    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.002448 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.002426    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.002928 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.002899    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.008228 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.008196    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.010930 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.010901    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.027688 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.027666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"]
Jul 02 17:05:49.029680 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:49.029655    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccf5c59_6b9e_4b6b_a99c_73abf094447a.slice/crio-9b55b439e7260003601f897b25c2b04692349013c36d4b8bc3b960cbddb6e674 WatchSource:0}: Error finding container 9b55b439e7260003601f897b25c2b04692349013c36d4b8bc3b960cbddb6e674: Status 404 returned error can't find the container with id 9b55b439e7260003601f897b25c2b04692349013c36d4b8bc3b960cbddb6e674
Jul 02 17:05:49.031100 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.031055    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:49.033280 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:05:49.033262    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a8bbb6_c410_4eef_a13d_c644159b8a0f.slice/crio-7e311816fb7d9302d664abb0f75ffc488637f75d644b15e9d1584c0e0d973fe1 WatchSource:0}: Error finding container 7e311816fb7d9302d664abb0f75ffc488637f75d644b15e9d1584c0e0d973fe1: Status 404 returned error can't find the container with id 7e311816fb7d9302d664abb0f75ffc488637f75d644b15e9d1584c0e0d973fe1
Jul 02 17:05:49.150837 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.150798    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7mcjd\" (UniqueName: \"kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd\") pod \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") "
Jul 02 17:05:49.151006 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.150907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28h7c\" (UniqueName: \"kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c\") pod \"bf688489-a09b-44b0-8f9c-ec0b01f8ff04\" (UID: \"bf688489-a09b-44b0-8f9c-ec0b01f8ff04\") "
Jul 02 17:05:49.151006 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.150981    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume\") pod \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\" (UID: \"e54a2614-7259-4a7b-9fd1-908b5f3a34ba\") "
Jul 02 17:05:49.151700 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.151668    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e54a2614-7259-4a7b-9fd1-908b5f3a34ba" (UID: "e54a2614-7259-4a7b-9fd1-908b5f3a34ba"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:05:49.153422 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.153397    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c" (OuterVolumeSpecName: "kube-api-access-28h7c") pod "bf688489-a09b-44b0-8f9c-ec0b01f8ff04" (UID: "bf688489-a09b-44b0-8f9c-ec0b01f8ff04"). InnerVolumeSpecName "kube-api-access-28h7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:05:49.153525 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.153424    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd" (OuterVolumeSpecName: "kube-api-access-7mcjd") pod "e54a2614-7259-4a7b-9fd1-908b5f3a34ba" (UID: "e54a2614-7259-4a7b-9fd1-908b5f3a34ba"). InnerVolumeSpecName "kube-api-access-7mcjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:05:49.154414 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.154384    2581 generic.go:356] "Generic (PLEG): container finished" podID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" containerID="7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb" exitCode=0
Jul 02 17:05:49.154497 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.154440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.154549 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.154483    2581 scope.go:117] "RemoveContainer" containerID="7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb"
Jul 02 17:05:49.155818 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.155796    2581 generic.go:356] "Generic (PLEG): container finished" podID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" containerID="a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc" exitCode=0
Jul 02 17:05:49.155928 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.155884    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.157419 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.157390    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.157793 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.157762    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6" event={"ID":"dccf5c59-6b9e-4b6b-a99c-73abf094447a","Type":"ContainerStarted","Data":"5e142933503ca5dbfb0a731cf5fb705923a0ea829f7784ccd0e8329740da1fd2"}
Jul 02 17:05:49.157938 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.157923    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:05:49.158014 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.157953    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6" event={"ID":"dccf5c59-6b9e-4b6b-a99c-73abf094447a","Type":"ContainerStarted","Data":"9b55b439e7260003601f897b25c2b04692349013c36d4b8bc3b960cbddb6e674"}
Jul 02 17:05:49.159708 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.159684    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" event={"ID":"82a8bbb6-c410-4eef-a13d-c644159b8a0f","Type":"ContainerStarted","Data":"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"}
Jul 02 17:05:49.159788 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.159729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" event={"ID":"82a8bbb6-c410-4eef-a13d-c644159b8a0f","Type":"ContainerStarted","Data":"7e311816fb7d9302d664abb0f75ffc488637f75d644b15e9d1584c0e0d973fe1"}
Jul 02 17:05:49.159788 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.159755    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:49.160631 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.160598    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.161414 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.161396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj" event={"ID":"1203da23-9b52-49f7-a0d1-e227e9655f67","Type":"ContainerStarted","Data":"14269e4c608417609b8c4857341e14a7979399c78392279be26bbc37cd8ee55a"}
Jul 02 17:05:49.164057 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.164030    2581 scope.go:117] "RemoveContainer" containerID="7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb"
Jul 02 17:05:49.164331 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.164299    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb\": container with ID starting with 7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb not found: ID does not exist" containerID="7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb"
Jul 02 17:05:49.164405 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.164342    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb"} err="failed to get container status \"7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb\": rpc error: code = NotFound desc = could not find container \"7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb\": container with ID starting with 7e1b038fb7d93aebb0c9b9afc1614729a2fcaaa626528cfccbe2d37e723cedfb not found: ID does not exist"
Jul 02 17:05:49.164405 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.164366    2581 scope.go:117] "RemoveContainer" containerID="a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc"
Jul 02 17:05:49.166240 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.166206    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:49.172135 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.172117    2581 scope.go:117] "RemoveContainer" containerID="a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc"
Jul 02 17:05:49.172384 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.172366    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc\": container with ID starting with a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc not found: ID does not exist" containerID="a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc"
Jul 02 17:05:49.172429 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.172392    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc"} err="failed to get container status \"a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc\": rpc error: code = NotFound desc = could not find container \"a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc\": container with ID starting with a7432487209d8f7a72681b0af4c0dd6d5301d7ef404f76c78d3777f7d438b7dc not found: ID does not exist"
Jul 02 17:05:49.199352 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.199305    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" podStartSLOduration=12.19928699 podStartE2EDuration="12.19928699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:49.199097567 +0000 UTC m=+540.886098547" watchObservedRunningTime="2026-07-02 17:05:49.19928699 +0000 UTC m=+540.886287963"
Jul 02 17:05:49.226411 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.226358    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6" podStartSLOduration=12.226343724 podStartE2EDuration="12.226343724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:49.225586459 +0000 UTC m=+540.912587432" watchObservedRunningTime="2026-07-02 17:05:49.226343724 +0000 UTC m=+540.913344697"
Jul 02 17:05:49.228489 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.228462    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.231155 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.231133    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-jlmvt\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-jlmvt"
Jul 02 17:05:49.252003 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.251973    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mcjd\" (UniqueName: \"kubernetes.io/projected/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-kube-api-access-7mcjd\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:05:49.252003 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.252003    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28h7c\" (UniqueName: \"kubernetes.io/projected/bf688489-a09b-44b0-8f9c-ec0b01f8ff04-kube-api-access-28h7c\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:05:49.252003 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.252017    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e54a2614-7259-4a7b-9fd1-908b5f3a34ba-extensions-socket-volume\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:05:49.263784 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:49.263737    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-25zsj" podStartSLOduration=1.523165227 podStartE2EDuration="23.263723973s" totalImagesPullingTime="21.740558746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:05:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:05:49.262620856 +0000 UTC m=+540.949621829" watchObservedRunningTime="2026-07-02 17:05:49.263723973 +0000 UTC m=+540.950724945"
Jul 02 17:05:49.265177 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:49.265151    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-sl4tm\" is forbidden: User \"system:node:ip-10-0-139-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-28.ec2.internal' and this object" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-sl4tm"
Jul 02 17:05:50.816194 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:50.816163    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf688489-a09b-44b0-8f9c-ec0b01f8ff04" path="/var/lib/kubelet/pods/bf688489-a09b-44b0-8f9c-ec0b01f8ff04/volumes"
Jul 02 17:05:50.816632 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:50.816511    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e54a2614-7259-4a7b-9fd1-908b5f3a34ba" path="/var/lib/kubelet/pods/e54a2614-7259-4a7b-9fd1-908b5f3a34ba/volumes"
Jul 02 17:05:53.918487 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:53.918454    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:53.918891 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:53.918625    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" podUID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" containerName="manager" containerID="cri-o://24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5" gracePeriod=10
Jul 02 17:05:54.169566 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.169509    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:54.187786 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.187749    2581 generic.go:356] "Generic (PLEG): container finished" podID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" containerID="24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5" exitCode=0
Jul 02 17:05:54.187944 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.187803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"
Jul 02 17:05:54.187944 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.187833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" event={"ID":"82a8bbb6-c410-4eef-a13d-c644159b8a0f","Type":"ContainerDied","Data":"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"}
Jul 02 17:05:54.187944 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.187906    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw" event={"ID":"82a8bbb6-c410-4eef-a13d-c644159b8a0f","Type":"ContainerDied","Data":"7e311816fb7d9302d664abb0f75ffc488637f75d644b15e9d1584c0e0d973fe1"}
Jul 02 17:05:54.187944 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.187931    2581 scope.go:117] "RemoveContainer" containerID="24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"
Jul 02 17:05:54.196639 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.196621    2581 scope.go:117] "RemoveContainer" containerID="24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"
Jul 02 17:05:54.196924 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:05:54.196900    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5\": container with ID starting with 24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5 not found: ID does not exist" containerID="24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"
Jul 02 17:05:54.196975 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.196937    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5"} err="failed to get container status \"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5\": rpc error: code = NotFound desc = could not find container \"24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5\": container with ID starting with 24f817ea97c47ce8ecd5260414fb02d71b43d1cab76b729f6a780e60e3dc5af5 not found: ID does not exist"
Jul 02 17:05:54.293983 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.293947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume\") pod \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") "
Jul 02 17:05:54.294178 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.293997    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjk5j\" (UniqueName: \"kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j\") pod \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\" (UID: \"82a8bbb6-c410-4eef-a13d-c644159b8a0f\") "
Jul 02 17:05:54.294369 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.294340    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "82a8bbb6-c410-4eef-a13d-c644159b8a0f" (UID: "82a8bbb6-c410-4eef-a13d-c644159b8a0f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:05:54.296108 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.296075    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j" (OuterVolumeSpecName: "kube-api-access-bjk5j") pod "82a8bbb6-c410-4eef-a13d-c644159b8a0f" (UID: "82a8bbb6-c410-4eef-a13d-c644159b8a0f"). InnerVolumeSpecName "kube-api-access-bjk5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:05:54.394593 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.394551    2581 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/82a8bbb6-c410-4eef-a13d-c644159b8a0f-extensions-socket-volume\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:05:54.394593 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.394586    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjk5j\" (UniqueName: \"kubernetes.io/projected/82a8bbb6-c410-4eef-a13d-c644159b8a0f-kube-api-access-bjk5j\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:05:54.523163 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.523133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:54.530943 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.530919    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7fksw"]
Jul 02 17:05:54.815941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:05:54.815915    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" path="/var/lib/kubelet/pods/82a8bbb6-c410-4eef-a13d-c644159b8a0f/volumes"
Jul 02 17:06:00.171167 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:00.171139    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nx9v6"
Jul 02 17:06:10.498061 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.498025    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"]
Jul 02 17:06:10.498444 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.498348    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" containerName="manager"
Jul 02 17:06:10.498444 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.498360    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" containerName="manager"
Jul 02 17:06:10.498444 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.498441    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="82a8bbb6-c410-4eef-a13d-c644159b8a0f" containerName="manager"
Jul 02 17:06:10.659845 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.659809    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"]
Jul 02 17:06:10.660048 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.659845    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.662447 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.662426    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 02 17:06:10.662611 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.662594    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-z8z44\""
Jul 02 17:06:10.727586 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727586 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727587    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkr8d\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-kube-api-access-kkr8d\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727723    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727768    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.727795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.728004 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727824    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.728004 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.727840    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.828961 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.828930    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.828961 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.828962    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.828985    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829064    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkr8d\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-kube-api-access-kkr8d\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829097    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829148    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829524 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829234    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829524 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829344    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829524 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829373    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829524 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829478    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829715 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829535    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.829884 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.829834    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.831440 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.831420    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.831519 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.831465    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.837847 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.837821    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.839126 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.839106    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkr8d\" (UniqueName: \"kubernetes.io/projected/2ea3470f-74ec-46fb-b2be-b88e9e0d4a47-kube-api-access-kkr8d\") pod \"maas-default-gateway-openshift-default-56654dcf7-62d7s\" (UID: \"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:10.969875 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:10.969832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:11.148541 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:11.148504    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"]
Jul 02 17:06:11.151727 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:06:11.151704    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea3470f_74ec_46fb_b2be_b88e9e0d4a47.slice/crio-fdcff3a2c4ee8536416cc825caf9837cd1b9d374d60adaece679adbbd1d456f8 WatchSource:0}: Error finding container fdcff3a2c4ee8536416cc825caf9837cd1b9d374d60adaece679adbbd1d456f8: Status 404 returned error can't find the container with id fdcff3a2c4ee8536416cc825caf9837cd1b9d374d60adaece679adbbd1d456f8
Jul 02 17:06:11.263624 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:11.263589    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" event={"ID":"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47","Type":"ContainerStarted","Data":"fdcff3a2c4ee8536416cc825caf9837cd1b9d374d60adaece679adbbd1d456f8"}
Jul 02 17:06:14.063978 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.063927    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:06:14.064317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.064014    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:06:14.064317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.064044    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:06:14.280646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.280604    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" event={"ID":"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47","Type":"ContainerStarted","Data":"6786c5ad5679f113bb8ea492de27269393542ca2d6641c339d480d9b95f34bdf"}
Jul 02 17:06:14.312283 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.312185    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" podStartSLOduration=1.40227842 podStartE2EDuration="4.312168766s" totalImagesPullingTime="2.909890346s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:06:14.309952249 +0000 UTC m=+565.996953236" watchObservedRunningTime="2026-07-02 17:06:14.312168766 +0000 UTC m=+565.999169737"
Jul 02 17:06:14.970705 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:14.970667    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:15.380148 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.380116    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:15.409345 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.409314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:15.409499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.409424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.412228 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.412204    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 02 17:06:15.471409 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.471377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4mc6\" (UniqueName: \"kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.471675 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.471653    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.477778 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.477752    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:15.573327 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.573294    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4mc6\" (UniqueName: \"kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.573508 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.573366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.574046 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.574023    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.584103 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.584078    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4mc6\" (UniqueName: \"kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6\") pod \"limitador-limitador-64c94fd64f-kg7nh\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.718953 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.718848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:15.798160 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.798125    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:06:15.942903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.942297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:15.942903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.942808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:06:15.942903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.942839    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:06:15.975617 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.975537    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:15.976816 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.976793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db3de8cc-7e73-4096-9b8d-f201abe807cb-config-file\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:15.976942 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:15.976823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jpmv\" (UniqueName: \"kubernetes.io/projected/db3de8cc-7e73-4096-9b8d-f201abe807cb-kube-api-access-8jpmv\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.018636 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.018593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:16.021444 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:06:16.021408    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd64afe1_7de2_44d3_b29b_9d51d015355a.slice/crio-0f66b9d423bfbae48ca3148bde36842abb3d31713670a0959c8c52e39558edb2 WatchSource:0}: Error finding container 0f66b9d423bfbae48ca3148bde36842abb3d31713670a0959c8c52e39558edb2: Status 404 returned error can't find the container with id 0f66b9d423bfbae48ca3148bde36842abb3d31713670a0959c8c52e39558edb2
Jul 02 17:06:16.077825 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.077798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db3de8cc-7e73-4096-9b8d-f201abe807cb-config-file\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.077982 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.077831    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jpmv\" (UniqueName: \"kubernetes.io/projected/db3de8cc-7e73-4096-9b8d-f201abe807cb-kube-api-access-8jpmv\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.078593 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.078573    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/db3de8cc-7e73-4096-9b8d-f201abe807cb-config-file\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.087464 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.087431    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jpmv\" (UniqueName: \"kubernetes.io/projected/db3de8cc-7e73-4096-9b8d-f201abe807cb-kube-api-access-8jpmv\") pod \"limitador-limitador-54d6d988d5-998h7\" (UID: \"db3de8cc-7e73-4096-9b8d-f201abe807cb\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.256399 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.256321    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:16.291393 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.291356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" event={"ID":"cd64afe1-7de2-44d3-b29b-9d51d015355a","Type":"ContainerStarted","Data":"0f66b9d423bfbae48ca3148bde36842abb3d31713670a0959c8c52e39558edb2"}
Jul 02 17:06:16.291695 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.291677    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:16.292748 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.292730    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:06:16.578304 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:16.578254    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:06:16.580900 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:06:16.580868    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb3de8cc_7e73_4096_9b8d_f201abe807cb.slice/crio-ceee0efa27484242fb9c22a5b37109bad33cdc1d9929c1bfc9868a90ea30af15 WatchSource:0}: Error finding container ceee0efa27484242fb9c22a5b37109bad33cdc1d9929c1bfc9868a90ea30af15: Status 404 returned error can't find the container with id ceee0efa27484242fb9c22a5b37109bad33cdc1d9929c1bfc9868a90ea30af15
Jul 02 17:06:17.302564 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:17.302495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7" event={"ID":"db3de8cc-7e73-4096-9b8d-f201abe807cb","Type":"ContainerStarted","Data":"ceee0efa27484242fb9c22a5b37109bad33cdc1d9929c1bfc9868a90ea30af15"}
Jul 02 17:06:20.318184 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.318147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7" event={"ID":"db3de8cc-7e73-4096-9b8d-f201abe807cb","Type":"ContainerStarted","Data":"2d97cb765fdf0341445b5cc7e0ff14378f4db32d3292a6016e63f6bb595fa150"}
Jul 02 17:06:20.318633 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.318316    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:20.319652 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.319625    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" event={"ID":"cd64afe1-7de2-44d3-b29b-9d51d015355a","Type":"ContainerStarted","Data":"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"}
Jul 02 17:06:20.319786 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.319675    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:20.338815 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.338766    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7" podStartSLOduration=1.99897466 podStartE2EDuration="5.338749745s" totalImagesPullingTime="3.339775085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:06:20.337206864 +0000 UTC m=+572.024207861" watchObservedRunningTime="2026-07-02 17:06:20.338749745 +0000 UTC m=+572.025750696"
Jul 02 17:06:20.359281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:20.359231    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" podStartSLOduration=1.467148658 podStartE2EDuration="5.359215755s" totalImagesPullingTime="3.892067097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:06:20.357687641 +0000 UTC m=+572.044688612" watchObservedRunningTime="2026-07-02 17:06:20.359215755 +0000 UTC m=+572.046216727"
Jul 02 17:06:31.326189 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:31.326161    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-998h7"
Jul 02 17:06:31.326566 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:31.326225    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:31.395498 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:31.395467    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:31.395660 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:31.395639    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" podUID="cd64afe1-7de2-44d3-b29b-9d51d015355a" containerName="limitador" containerID="cri-o://ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350" gracePeriod=30
Jul 02 17:06:31.950253 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:31.950231    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:32.016089 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.016005    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file\") pod \"cd64afe1-7de2-44d3-b29b-9d51d015355a\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") "
Jul 02 17:06:32.016226 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.016117    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4mc6\" (UniqueName: \"kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6\") pod \"cd64afe1-7de2-44d3-b29b-9d51d015355a\" (UID: \"cd64afe1-7de2-44d3-b29b-9d51d015355a\") "
Jul 02 17:06:32.016413 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.016392    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file" (OuterVolumeSpecName: "config-file") pod "cd64afe1-7de2-44d3-b29b-9d51d015355a" (UID: "cd64afe1-7de2-44d3-b29b-9d51d015355a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 02 17:06:32.018138 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.018114    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6" (OuterVolumeSpecName: "kube-api-access-g4mc6") pod "cd64afe1-7de2-44d3-b29b-9d51d015355a" (UID: "cd64afe1-7de2-44d3-b29b-9d51d015355a"). InnerVolumeSpecName "kube-api-access-g4mc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:06:32.117299 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.117266    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4mc6\" (UniqueName: \"kubernetes.io/projected/cd64afe1-7de2-44d3-b29b-9d51d015355a-kube-api-access-g4mc6\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:06:32.117299 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.117294    2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cd64afe1-7de2-44d3-b29b-9d51d015355a-config-file\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:06:32.384129 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.384098    2581 generic.go:356] "Generic (PLEG): container finished" podID="cd64afe1-7de2-44d3-b29b-9d51d015355a" containerID="ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350" exitCode=0
Jul 02 17:06:32.384503 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.384160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"
Jul 02 17:06:32.384503 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.384185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" event={"ID":"cd64afe1-7de2-44d3-b29b-9d51d015355a","Type":"ContainerDied","Data":"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"}
Jul 02 17:06:32.384503 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.384271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-kg7nh" event={"ID":"cd64afe1-7de2-44d3-b29b-9d51d015355a","Type":"ContainerDied","Data":"0f66b9d423bfbae48ca3148bde36842abb3d31713670a0959c8c52e39558edb2"}
Jul 02 17:06:32.384503 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.384299    2581 scope.go:117] "RemoveContainer" containerID="ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"
Jul 02 17:06:32.392771 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.392752    2581 scope.go:117] "RemoveContainer" containerID="ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"
Jul 02 17:06:32.393032 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:06:32.393016    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350\": container with ID starting with ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350 not found: ID does not exist" containerID="ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"
Jul 02 17:06:32.393090 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.393038    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350"} err="failed to get container status \"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350\": rpc error: code = NotFound desc = could not find container \"ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350\": container with ID starting with ed7ec7899daa1ce62c735929d713f86a6d2fa84c98585bae9993bcfeda680350 not found: ID does not exist"
Jul 02 17:06:32.406001 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.405978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:32.408835 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.408814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-kg7nh"]
Jul 02 17:06:32.815604 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:32.815575    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd64afe1-7de2-44d3-b29b-9d51d015355a" path="/var/lib/kubelet/pods/cd64afe1-7de2-44d3-b29b-9d51d015355a/volumes"
Jul 02 17:06:48.780800 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:48.780774    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:06:48.785430 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:48.785410    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:06:59.191706 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.191671    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"]
Jul 02 17:06:59.192280 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.192035    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd64afe1-7de2-44d3-b29b-9d51d015355a" containerName="limitador"
Jul 02 17:06:59.192280 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.192047    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd64afe1-7de2-44d3-b29b-9d51d015355a" containerName="limitador"
Jul 02 17:06:59.192280 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.192113    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd64afe1-7de2-44d3-b29b-9d51d015355a" containerName="limitador"
Jul 02 17:06:59.199232 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.199214    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"
Jul 02 17:06:59.202191 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.202172    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"openshift-service-ca.crt\""
Jul 02 17:06:59.203457 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.203428    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"keycloak-system\"/\"keycloak-operator-dockercfg-lq2fh\""
Jul 02 17:06:59.203457 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.203449    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"keycloak-system\"/\"kube-root-ca.crt\""
Jul 02 17:06:59.205643 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.205621    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"]
Jul 02 17:06:59.235531 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.235504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpk5\" (UniqueName: \"kubernetes.io/projected/f54f8d44-30a2-448b-a9d2-c48615e25301-kube-api-access-rbpk5\") pod \"keycloak-operator-d8b9d7b4d-kzwfv\" (UID: \"f54f8d44-30a2-448b-a9d2-c48615e25301\") " pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"
Jul 02 17:06:59.336128 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.336094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpk5\" (UniqueName: \"kubernetes.io/projected/f54f8d44-30a2-448b-a9d2-c48615e25301-kube-api-access-rbpk5\") pod \"keycloak-operator-d8b9d7b4d-kzwfv\" (UID: \"f54f8d44-30a2-448b-a9d2-c48615e25301\") " pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"
Jul 02 17:06:59.346216 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.346189    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpk5\" (UniqueName: \"kubernetes.io/projected/f54f8d44-30a2-448b-a9d2-c48615e25301-kube-api-access-rbpk5\") pod \"keycloak-operator-d8b9d7b4d-kzwfv\" (UID: \"f54f8d44-30a2-448b-a9d2-c48615e25301\") " pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"
Jul 02 17:06:59.509790 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.509714    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"
Jul 02 17:06:59.652504 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:06:59.652480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv"]
Jul 02 17:06:59.655353 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:06:59.655321    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54f8d44_30a2_448b_a9d2_c48615e25301.slice/crio-d3e1d4c10d52e2b6e69ecdbd8e599e17240e854f1ba5e4810a44b8ed0e367038 WatchSource:0}: Error finding container d3e1d4c10d52e2b6e69ecdbd8e599e17240e854f1ba5e4810a44b8ed0e367038: Status 404 returned error can't find the container with id d3e1d4c10d52e2b6e69ecdbd8e599e17240e854f1ba5e4810a44b8ed0e367038
Jul 02 17:07:00.514043 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:07:00.513967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv" event={"ID":"f54f8d44-30a2-448b-a9d2-c48615e25301","Type":"ContainerStarted","Data":"d3e1d4c10d52e2b6e69ecdbd8e599e17240e854f1ba5e4810a44b8ed0e367038"}
Jul 02 17:07:05.541902 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:07:05.541851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv" event={"ID":"f54f8d44-30a2-448b-a9d2-c48615e25301","Type":"ContainerStarted","Data":"60a77281f549463cbac52ab2fbc7da66e48ffde2938d5b721ac2c79b60b94d16"}
Jul 02 17:07:05.572459 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:07:05.572401    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="keycloak-system/keycloak-operator-d8b9d7b4d-kzwfv" podStartSLOduration=1.061762719 podStartE2EDuration="6.572385676s" totalImagesPullingTime="5.510622957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:06:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:07:05.571219351 +0000 UTC m=+617.258220325" watchObservedRunningTime="2026-07-02 17:07:05.572385676 +0000 UTC m=+617.259386650"
Jul 02 17:08:26.871917 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:26.871878    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:08:34.943595 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.943563    2581 generic.go:356] "Generic (PLEG): container finished" podID="2ea3470f-74ec-46fb-b2be-b88e9e0d4a47" containerID="6786c5ad5679f113bb8ea492de27269393542ca2d6641c339d480d9b95f34bdf" exitCode=0
Jul 02 17:08:34.944052 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.943640    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" event={"ID":"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47","Type":"ContainerDied","Data":"6786c5ad5679f113bb8ea492de27269393542ca2d6641c339d480d9b95f34bdf"}
Jul 02 17:08:34.944097 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.944063    2581 scope.go:117] "RemoveContainer" containerID="6786c5ad5679f113bb8ea492de27269393542ca2d6641c339d480d9b95f34bdf"
Jul 02 17:08:34.944942 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.944915    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:08:34.945006 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.944979    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:08:34.945043 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:34.945009    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 02 17:08:35.954502 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:35.954460    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" event={"ID":"2ea3470f-74ec-46fb-b2be-b88e9e0d4a47","Type":"ContainerStarted","Data":"f17c52d2e4bb8f6e89055967c9879ed2101ebde4af58c39215d042f3fe371ec8"}
Jul 02 17:08:35.970109 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:35.970081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:08:36.970619 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:36.970585    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62d7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 02 17:08:36.971013 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:36.970637    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" podUID="2ea3470f-74ec-46fb-b2be-b88e9e0d4a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 02 17:08:37.971122 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:37.971089    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62d7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 02 17:08:37.971502 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:37.971172    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" podUID="2ea3470f-74ec-46fb-b2be-b88e9e0d4a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 02 17:08:38.970244 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:38.970207    2581 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-62d7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Jul 02 17:08:38.970422 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:38.970258    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s" podUID="2ea3470f-74ec-46fb-b2be-b88e9e0d4a47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Jul 02 17:08:39.974310 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:39.974281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:08:39.974734 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:39.974550    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:08:39.975142 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:08:39.975125    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-62d7s"
Jul 02 17:09:26.021555 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:26.021518    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:09:34.422899 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:34.422843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:09:47.701800 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.701753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"]
Jul 02 17:09:47.705302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.705280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.708269 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.708246    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 02 17:09:47.709514 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.709485    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 02 17:09:47.709615 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.709523    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5lqpq\""
Jul 02 17:09:47.709615 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.709538    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 02 17:09:47.716928 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.716905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"]
Jul 02 17:09:47.837907 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.837846    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6q5g\" (UniqueName: \"kubernetes.io/projected/a7607823-7780-46de-9382-b4de13c449cf-kube-api-access-p6q5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.838079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.837929    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7607823-7780-46de-9382-b4de13c449cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.838079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.837951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.838079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.837967    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.838079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.837997    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.838079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.838019    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938482 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938442    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7607823-7780-46de-9382-b4de13c449cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938482 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938482    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938695 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938508    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938695 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938695 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938687    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.938843 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.938740    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6q5g\" (UniqueName: \"kubernetes.io/projected/a7607823-7780-46de-9382-b4de13c449cf-kube-api-access-p6q5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.939053 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.939030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.939123 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.939051    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.939123 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.939094    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.940895 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.940848    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a7607823-7780-46de-9382-b4de13c449cf-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.941055 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.941039    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7607823-7780-46de-9382-b4de13c449cf-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:47.947010 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:47.946986    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6q5g\" (UniqueName: \"kubernetes.io/projected/a7607823-7780-46de-9382-b4de13c449cf-kube-api-access-p6q5g\") pod \"e2e-trlp-test-simulated-kserve-598466548d-h2j8z\" (UID: \"a7607823-7780-46de-9382-b4de13c449cf\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:48.017768 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:48.017671    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:48.154041 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:48.153944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"]
Jul 02 17:09:48.156799 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:09:48.156773    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7607823_7780_46de_9382_b4de13c449cf.slice/crio-bcce56333e3cb14de2996c218d7bea7c7f44a5663fb959ce9a367f7b22ddd16d WatchSource:0}: Error finding container bcce56333e3cb14de2996c218d7bea7c7f44a5663fb959ce9a367f7b22ddd16d: Status 404 returned error can't find the container with id bcce56333e3cb14de2996c218d7bea7c7f44a5663fb959ce9a367f7b22ddd16d
Jul 02 17:09:48.158246 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:48.158224    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 17:09:48.273738 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:48.273652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z" event={"ID":"a7607823-7780-46de-9382-b4de13c449cf","Type":"ContainerStarted","Data":"bcce56333e3cb14de2996c218d7bea7c7f44a5663fb959ce9a367f7b22ddd16d"}
Jul 02 17:09:48.917148 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:48.917113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:09:50.288068 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:50.288029    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z" event={"ID":"a7607823-7780-46de-9382-b4de13c449cf","Type":"ContainerStarted","Data":"7841db1b6730a9490c3e8fa5c7890226fe27b47c86d4a3112167427130e97083"}
Jul 02 17:09:50.305571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:50.305519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z" podStartSLOduration=2.213579153 podStartE2EDuration="3.305502033s" totalImagesPullingTime="1.09192288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:09:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:09:50.304637586 +0000 UTC m=+781.991638556" watchObservedRunningTime="2026-07-02 17:09:50.305502033 +0000 UTC m=+781.992503011"
Jul 02 17:09:58.017935 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.017893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:58.018447 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.017981    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:58.030253 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.030228    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:58.191256 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.191218    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"]
Jul 02 17:09:58.194883 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.194849    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.197841 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.197824    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 02 17:09:58.209109 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.209089    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"]
Jul 02 17:09:58.325941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.325910    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp5x2\" (UniqueName: \"kubernetes.io/projected/19195264-789b-41e5-a242-ad9c4665afc7-kube-api-access-lp5x2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.326134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.325948    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.326134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.326017    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.326134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.326044    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.326134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.326069    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.326134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.326111    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19195264-789b-41e5-a242-ad9c4665afc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.335769 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.335745    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-h2j8z"
Jul 02 17:09:58.427490 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427677 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427677 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427533    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427677 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427594    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19195264-789b-41e5-a242-ad9c4665afc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427677 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp5x2\" (UniqueName: \"kubernetes.io/projected/19195264-789b-41e5-a242-ad9c4665afc7-kube-api-access-lp5x2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427957 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.427957 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427922    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.428036 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.427972    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.428171 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.428147    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.429772 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.429751    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19195264-789b-41e5-a242-ad9c4665afc7-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.430276 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.430254    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19195264-789b-41e5-a242-ad9c4665afc7-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.438623 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.438602    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp5x2\" (UniqueName: \"kubernetes.io/projected/19195264-789b-41e5-a242-ad9c4665afc7-kube-api-access-lp5x2\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq\" (UID: \"19195264-789b-41e5-a242-ad9c4665afc7\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.504274 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.504238    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:09:58.635321 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:58.635293    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"]
Jul 02 17:09:58.636654 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:09:58.636626    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19195264_789b_41e5_a242_ad9c4665afc7.slice/crio-fa84a0d449d5ff66d94a692c36fad2bb21474c251e7e568b586d30acf90fbf93 WatchSource:0}: Error finding container fa84a0d449d5ff66d94a692c36fad2bb21474c251e7e568b586d30acf90fbf93: Status 404 returned error can't find the container with id fa84a0d449d5ff66d94a692c36fad2bb21474c251e7e568b586d30acf90fbf93
Jul 02 17:09:59.330256 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:59.330215    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq" event={"ID":"19195264-789b-41e5-a242-ad9c4665afc7","Type":"ContainerStarted","Data":"b016bbb20f839e2c5de7f449cc73943180c5bfe53d6e8ee92cf1bfdd21a9f8e8"}
Jul 02 17:09:59.330719 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:59.330277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq" event={"ID":"19195264-789b-41e5-a242-ad9c4665afc7","Type":"ContainerStarted","Data":"fa84a0d449d5ff66d94a692c36fad2bb21474c251e7e568b586d30acf90fbf93"}
Jul 02 17:09:59.349307 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:59.349254    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq" podStartSLOduration=1.086586102 podStartE2EDuration="1.34923965s" totalImagesPullingTime="262.653548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:09:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:09:59.347567603 +0000 UTC m=+791.034568595" watchObservedRunningTime="2026-07-02 17:09:59.34923965 +0000 UTC m=+791.036240699"
Jul 02 17:09:59.715740 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:09:59.715661    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:10:08.504551 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:08.504510    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:10:08.504963 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:08.504573    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:10:08.517148 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:08.517120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:10:09.387024 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:09.386993    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq"
Jul 02 17:10:16.815885 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:16.815801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:10:28.288200 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.288167    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"]
Jul 02 17:10:28.295070 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.295040    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.297758 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.297739    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 02 17:10:28.303898 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.303874    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"]
Jul 02 17:10:28.375338 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375309    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.375338 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375342    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b267\" (UniqueName: \"kubernetes.io/projected/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-kube-api-access-8b267\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.375557 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.375557 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.375557 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.375557 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.375513    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476506 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476689 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476689 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b267\" (UniqueName: \"kubernetes.io/projected/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-kube-api-access-8b267\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476689 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476689 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.476689 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.476614    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.477070 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.477042    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.477191 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.477090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.477191 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.477091    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.478870 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.478837    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.479258 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.479238    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.484602 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.484577    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b267\" (UniqueName: \"kubernetes.io/projected/e60974ae-4bbc-4f5c-89c0-9d5cafda1498-kube-api-access-8b267\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j\" (UID: \"e60974ae-4bbc-4f5c-89c0-9d5cafda1498\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.605342 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.605308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:28.747567 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:28.747541    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"]
Jul 02 17:10:28.750271 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:10:28.750244    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode60974ae_4bbc_4f5c_89c0_9d5cafda1498.slice/crio-73ee190b21e01344426447921c3751e3d7bf5e2f313b5cff226e84d2382f35a9 WatchSource:0}: Error finding container 73ee190b21e01344426447921c3751e3d7bf5e2f313b5cff226e84d2382f35a9: Status 404 returned error can't find the container with id 73ee190b21e01344426447921c3751e3d7bf5e2f313b5cff226e84d2382f35a9
Jul 02 17:10:29.469455 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:29.469420    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j" event={"ID":"e60974ae-4bbc-4f5c-89c0-9d5cafda1498","Type":"ContainerStarted","Data":"bdca3aa4e1eddde4699e411ede2ea303dcf78216ed378cd31eea06f430e47500"}
Jul 02 17:10:29.470134 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:29.469462    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j" event={"ID":"e60974ae-4bbc-4f5c-89c0-9d5cafda1498","Type":"ContainerStarted","Data":"73ee190b21e01344426447921c3751e3d7bf5e2f313b5cff226e84d2382f35a9"}
Jul 02 17:10:29.486600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:29.486547    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j" podStartSLOduration=1.245550627 podStartE2EDuration="1.486529817s" totalImagesPullingTime="240.97919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:10:29.486394233 +0000 UTC m=+821.173395219" watchObservedRunningTime="2026-07-02 17:10:29.486529817 +0000 UTC m=+821.173530790"
Jul 02 17:10:30.713984 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:30.713948    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:10:38.605599 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:38.605560    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:38.606037 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:38.605616    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:38.617966 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:38.617940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:10:39.529463 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:10:39.529432    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j"
Jul 02 17:11:48.809329 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:11:48.809250    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:11:48.812734 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:11:48.812710    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:12:08.115484 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:12:08.115447    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:12:17.414996 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:12:17.414961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:12:29.618703 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:12:29.618658    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:12:37.523221 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:12:37.523185    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:12:48.615069 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:12:48.615033    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:13:16.314390 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:13:16.314301    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:13:27.114385 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:13:27.114348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:13:59.911382 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:13:59.911348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:09.409514 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:09.409480    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:18.021920 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:18.021881    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:26.908309 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:26.908267    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:37.016100 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:37.016062    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:47.720984 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:47.720903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:14:58.527745 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:14:58.527712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:15:08.217084 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:15:08.217046    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:15:31.718272 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:15:31.718232    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:15:40.706022 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:15:40.705985    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:16:05.221171 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:05.221132    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:16:15.412475 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:15.412397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:16:31.520099 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:31.520066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:16:42.513368 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:42.513331    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:16:48.839248 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:48.839216    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:16:48.840259 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:48.840233    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:16:51.813294 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:16:51.813261    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:17:01.219534 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:17:01.219490    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:17:11.110872 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:17:11.110830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:00.516274 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:00.516237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:09.909761 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:09.909719    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:22.718638 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:22.718557    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:34.218000 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:34.217963    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:46.101572 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:46.101534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:19:56.314132 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:19:56.314099    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:20:05.312249 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:20:05.312213    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:20:16.107196 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:20:16.107164    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:20:27.104690 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:20:27.104650    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:20:36.106670 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:20:36.106634    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:20:46.414029 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:20:46.413952    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:21:11.020710 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:11.020675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:21:18.816991 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:18.816961    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:21:28.716017 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:28.715984    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:21:40.709097 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:40.709064    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:21:48.867341 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:48.867309    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:21:48.867773 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:48.867745    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:21:57.117995 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:21:57.117947    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:22:09.109733 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:22:09.109697    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:22:18.608204 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:22:18.608113    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:22:26.611168 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:22:26.611133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:22:37.710194 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:22:37.710151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:22:47.112977 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:22:47.112932    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:00.410370 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:00.410328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:09.819906 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:09.819846    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:19.316909 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:19.316872    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:28.415931 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:28.415893    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:42.509010 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:42.508928    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:23:55.517611 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:23:55.517577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:24:11.411069 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:24:11.411023    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:24:25.215012 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:24:25.214981    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:24:38.406753 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:24:38.406712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:24:50.619116 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:24:50.619080    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:25:00.208313 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:25:00.208276    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:25:08.185273 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:25:08.185237    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:25:31.421752 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:25:31.421712    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:25:40.316655 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:25:40.316605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:26:48.893449 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:26:48.893420    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:26:48.895228 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:26:48.895200    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:27:17.223923 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:27:17.223883    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:27:26.208239 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:27:26.208199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:27:36.006700 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:27:36.006665    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:27:47.128566 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:27:47.128526    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:28:17.905802 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:17.905726    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:28:30.086479 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.086442    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:28:30.089642 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.089626    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.093524 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.093494    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ca038502\"/\"e2e-lifecycle-model-ca038502-kserve-self-signed-certs\""
Jul 02 17:28:30.093654 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.093499    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ca038502\"/\"openshift-service-ca.crt\""
Jul 02 17:28:30.093654 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.093499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-ca038502\"/\"default-dockercfg-4dwxl\""
Jul 02 17:28:30.093654 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.093573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-ca038502\"/\"kube-root-ca.crt\""
Jul 02 17:28:30.097885 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.097850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:28:30.131238 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131197    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2ms5\" (UniqueName: \"kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.131349 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131239    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.131349 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131271    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.131349 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131306    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.131514 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131394    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.131514 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.131430    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232253 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232253 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232471 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232432    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2ms5\" (UniqueName: \"kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232471 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232579 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232481    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232579 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232495    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232685 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232603    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232815 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.232904 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.232830    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.234678 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.234656    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.234814 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.234804    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.240479 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.240452    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2ms5\" (UniqueName: \"kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5\") pod \"e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") " pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.399737 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.399658    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:30.529683 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:30.529660    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:28:30.531326 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:28:30.531294    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod791edd22_bc19_49d9_b8c5_158d15642354.slice/crio-cbf147f7dfaa5575c2fa7aaf77e4f75759a23dda733692d105982ff739a9a569 WatchSource:0}: Error finding container cbf147f7dfaa5575c2fa7aaf77e4f75759a23dda733692d105982ff739a9a569: Status 404 returned error can't find the container with id cbf147f7dfaa5575c2fa7aaf77e4f75759a23dda733692d105982ff739a9a569
Jul 02 17:28:31.483418 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:31.483385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" event={"ID":"791edd22-bc19-49d9-b8c5-158d15642354","Type":"ContainerStarted","Data":"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"}
Jul 02 17:28:31.483418 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:31.483425    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" event={"ID":"791edd22-bc19-49d9-b8c5-158d15642354","Type":"ContainerStarted","Data":"cbf147f7dfaa5575c2fa7aaf77e4f75759a23dda733692d105982ff739a9a569"}
Jul 02 17:28:31.504189 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:31.504145    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" podStartSLOduration=1.504130322 podStartE2EDuration="1.504130322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:28:31.503763025 +0000 UTC m=+1903.190764024" watchObservedRunningTime="2026-07-02 17:28:31.504130322 +0000 UTC m=+1903.191131293"
Jul 02 17:28:33.401927 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:33.401894    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:28:40.400434 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:40.400401    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:40.400968 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:40.400468    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:40.412851 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:40.412830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:40.537725 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:40.537697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:28:48.206499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:48.206463    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:28:58.218046 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:28:58.218007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:29:04.861201 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:04.861168    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:29:08.212299 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:08.212259    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:29:19.881478 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:19.881405    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" podUID="791edd22-bc19-49d9-b8c5-158d15642354" containerName="main" containerID="cri-o://7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5" gracePeriod=45
Jul 02 17:29:20.131631 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.131567    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:29:20.161676 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161651    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k2ms5\" (UniqueName: \"kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.161838 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161707    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.161838 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.161838 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161764    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.161838 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161780    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.161838 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.161830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm\") pod \"791edd22-bc19-49d9-b8c5-158d15642354\" (UID: \"791edd22-bc19-49d9-b8c5-158d15642354\") "
Jul 02 17:29:20.163032 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.162262    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache" (OuterVolumeSpecName: "model-cache") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:29:20.163032 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.162548    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:29:20.163032 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.162604    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home" (OuterVolumeSpecName: "home") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:29:20.164419 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.164396    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm" (OuterVolumeSpecName: "dshm") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:29:20.164621 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.164584    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5" (OuterVolumeSpecName: "kube-api-access-k2ms5") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "kube-api-access-k2ms5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:29:20.164783 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.164755    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "791edd22-bc19-49d9-b8c5-158d15642354" (UID: "791edd22-bc19-49d9-b8c5-158d15642354"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:29:20.263204 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263178    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.263204 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263200    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/791edd22-bc19-49d9-b8c5-158d15642354-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.263204 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263209    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.263403 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263217    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.263403 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263224    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/791edd22-bc19-49d9-b8c5-158d15642354-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.263403 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.263232    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2ms5\" (UniqueName: \"kubernetes.io/projected/791edd22-bc19-49d9-b8c5-158d15642354-kube-api-access-k2ms5\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:29:20.714140 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.714104    2581 generic.go:356] "Generic (PLEG): container finished" podID="791edd22-bc19-49d9-b8c5-158d15642354" containerID="7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5" exitCode=0
Jul 02 17:29:20.714302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.714144    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"
Jul 02 17:29:20.714302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.714170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" event={"ID":"791edd22-bc19-49d9-b8c5-158d15642354","Type":"ContainerDied","Data":"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"}
Jul 02 17:29:20.714302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.714204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2" event={"ID":"791edd22-bc19-49d9-b8c5-158d15642354","Type":"ContainerDied","Data":"cbf147f7dfaa5575c2fa7aaf77e4f75759a23dda733692d105982ff739a9a569"}
Jul 02 17:29:20.714302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.714222    2581 scope.go:117] "RemoveContainer" containerID="7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"
Jul 02 17:29:20.723027 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.723011    2581 scope.go:117] "RemoveContainer" containerID="7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"
Jul 02 17:29:20.723269 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:29:20.723253    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5\": container with ID starting with 7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5 not found: ID does not exist" containerID="7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"
Jul 02 17:29:20.723343 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.723277    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5"} err="failed to get container status \"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5\": rpc error: code = NotFound desc = could not find container \"7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5\": container with ID starting with 7e240f2aac791a4bcd3205aa001ee8af293755a50044c1be82a3a1c857c83aa5 not found: ID does not exist"
Jul 02 17:29:20.734882 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.734847    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:29:20.739102 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.739084    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-ca038502/e2e-lifecycle-model-ca038502-kserve-5f475d6d8c-7mgt2"]
Jul 02 17:29:20.819263 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:20.818599    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="791edd22-bc19-49d9-b8c5-158d15642354" path="/var/lib/kubelet/pods/791edd22-bc19-49d9-b8c5-158d15642354/volumes"
Jul 02 17:29:45.107674 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:45.107593    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:29:57.509112 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:29:57.509066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:30:11.215440 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:30:11.215402    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:30:56.410787 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:30:56.410751    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:31:07.807868 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:07.807819    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:31:19.396394 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.396307    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:31:19.396762 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.396685    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="791edd22-bc19-49d9-b8c5-158d15642354" containerName="main"
Jul 02 17:31:19.396762 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.396697    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="791edd22-bc19-49d9-b8c5-158d15642354" containerName="main"
Jul 02 17:31:19.396850 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.396773    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="791edd22-bc19-49d9-b8c5-158d15642354" containerName="main"
Jul 02 17:31:19.399703 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.399688    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.402304 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.402286    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8a6030\"/\"openshift-service-ca.crt\""
Jul 02 17:31:19.403508 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.403492    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8a6030\"/\"default-dockercfg-s9lbc\""
Jul 02 17:31:19.403577 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.403527    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8a6030\"/\"test-model-8a6030-kserve-self-signed-certs\""
Jul 02 17:31:19.403577 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.403563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8a6030\"/\"kube-root-ca.crt\""
Jul 02 17:31:19.407208 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.407165    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:31:19.433896 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.433844    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.434048 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.433918    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.434048 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.433969    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.434048 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.433992    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.434196 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.434045    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksjv5\" (UniqueName: \"kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.434196 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.434084    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535467 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535425    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksjv5\" (UniqueName: \"kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535673 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535478    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535673 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535673 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535846 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.535846 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.535752    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.536036 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.536017    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.536113 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.536054    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.536113 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.536083    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.537817 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.537795    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.538163 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.538143    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.544445 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.544423    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksjv5\" (UniqueName: \"kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5\") pod \"test-model-8a6030-kserve-84f57d4db9-kxvxw\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") " pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.709366 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.709280    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:19.846125 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:19.846100    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:31:19.847953 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:31:19.847924    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfc1665b7_e923_4048_b181_dc6383ccebb2.slice/crio-3fbf5f71b22db0a224d62fbe794669ce32736aa712dad63192a6bda46abe331f WatchSource:0}: Error finding container 3fbf5f71b22db0a224d62fbe794669ce32736aa712dad63192a6bda46abe331f: Status 404 returned error can't find the container with id 3fbf5f71b22db0a224d62fbe794669ce32736aa712dad63192a6bda46abe331f
Jul 02 17:31:20.268895 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:20.268848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" event={"ID":"fc1665b7-e923-4048-b181-dc6383ccebb2","Type":"ContainerStarted","Data":"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"}
Jul 02 17:31:20.268895 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:20.268900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" event={"ID":"fc1665b7-e923-4048-b181-dc6383ccebb2","Type":"ContainerStarted","Data":"3fbf5f71b22db0a224d62fbe794669ce32736aa712dad63192a6bda46abe331f"}
Jul 02 17:31:20.285587 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:20.285540    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" podStartSLOduration=1.285524514 podStartE2EDuration="1.285524514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:31:20.284686697 +0000 UTC m=+2071.971687672" watchObservedRunningTime="2026-07-02 17:31:20.285524514 +0000 UTC m=+2071.972525486"
Jul 02 17:31:22.616890 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:22.616841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:31:29.709630 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:29.709589    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:29.709630 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:29.709636    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:29.721731 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:29.721706    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:30.326262 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:30.326232    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:31:35.706819 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:35.706782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:31:48.922283 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:48.922242    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:31:48.925074 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:48.925054    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:31:57.295725 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.295692    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:31:57.298941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.298923    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.301767 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.301745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5569c1\"/\"openshift-service-ca.crt\""
Jul 02 17:31:57.303120 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.303094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5569c1\"/\"test-model-5569c1-kserve-self-signed-certs\""
Jul 02 17:31:57.303236 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.303161    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-5569c1\"/\"kube-root-ca.crt\""
Jul 02 17:31:57.303236 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.303163    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-5569c1\"/\"default-dockercfg-jp8k4\""
Jul 02 17:31:57.310835 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.310814    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:31:57.454528 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454492    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.454528 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgv5d\" (UniqueName: \"kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.454778 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454561    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.454778 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454651    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.454778 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.454778 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.454743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.555834 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555743    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.555834 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555798    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.555834 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555837    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556130 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555883    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556130 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555935    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556130 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.555950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgv5d\" (UniqueName: \"kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556285 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.556235    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556285 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.556247    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.556391 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.556374    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.558187 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.558162    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.558459 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.558441    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.564085 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.564062    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgv5d\" (UniqueName: \"kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d\") pod \"test-model-5569c1-kserve-8f495dd79-djl82\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") " pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.608038 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.608012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:31:57.737791 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:57.737763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:31:57.739697 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:31:57.739665    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32e0f1d9_e00a_458d_9593_abefdf4c5702.slice/crio-1e9e5593285a95062b3245ef2b7efa1d7a2b0da46b5aff46973dd975d7a87844 WatchSource:0}: Error finding container 1e9e5593285a95062b3245ef2b7efa1d7a2b0da46b5aff46973dd975d7a87844: Status 404 returned error can't find the container with id 1e9e5593285a95062b3245ef2b7efa1d7a2b0da46b5aff46973dd975d7a87844
Jul 02 17:31:58.449911 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:58.449877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" event={"ID":"32e0f1d9-e00a-458d-9593-abefdf4c5702","Type":"ContainerStarted","Data":"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"}
Jul 02 17:31:58.449911 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:58.449917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" event={"ID":"32e0f1d9-e00a-458d-9593-abefdf4c5702","Type":"ContainerStarted","Data":"1e9e5593285a95062b3245ef2b7efa1d7a2b0da46b5aff46973dd975d7a87844"}
Jul 02 17:31:58.470295 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:31:58.470255    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" podStartSLOduration=1.4702405920000001 podStartE2EDuration="1.470240592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:31:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:31:58.468181076 +0000 UTC m=+2110.155182060" watchObservedRunningTime="2026-07-02 17:31:58.470240592 +0000 UTC m=+2110.157241565"
Jul 02 17:32:01.631051 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:01.631016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:32:07.609220 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:07.609178    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:07.609600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:07.609245    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:07.621431 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:07.621403    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:08.507108 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:08.507080    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:14.608275 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:14.608234    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:32:17.121664 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:17.121630    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:32:18.096541 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:18.096507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:32:29.607794 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:29.607753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:32:32.141430 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.141387    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" podUID="fc1665b7-e923-4048-b181-dc6383ccebb2" containerName="main" containerID="cri-o://3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc" gracePeriod=45
Jul 02 17:32:32.392988 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.392930    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:32:32.455787 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.455745    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.455959 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.455840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksjv5\" (UniqueName: \"kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.456002 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.455988    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.456045 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456031    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.456084 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456059    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.456117 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456107    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache\") pod \"fc1665b7-e923-4048-b181-dc6383ccebb2\" (UID: \"fc1665b7-e923-4048-b181-dc6383ccebb2\") "
Jul 02 17:32:32.456359 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456299    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home" (OuterVolumeSpecName: "home") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:32.456359 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456313    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:32.456537 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456455    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.456537 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456471    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache" (OuterVolumeSpecName: "model-cache") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:32.456537 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.456475    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.458123 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.458099    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm" (OuterVolumeSpecName: "dshm") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:32.458370 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.458347    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5" (OuterVolumeSpecName: "kube-api-access-ksjv5") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "kube-api-access-ksjv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:32:32.458370 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.458346    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fc1665b7-e923-4048-b181-dc6383ccebb2" (UID: "fc1665b7-e923-4048-b181-dc6383ccebb2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:32:32.557424 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.557382    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.557424 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.557421    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fc1665b7-e923-4048-b181-dc6383ccebb2-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.557673 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.557437    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fc1665b7-e923-4048-b181-dc6383ccebb2-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.557673 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.557448    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksjv5\" (UniqueName: \"kubernetes.io/projected/fc1665b7-e923-4048-b181-dc6383ccebb2-kube-api-access-ksjv5\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:32.612695 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.612661    2581 generic.go:356] "Generic (PLEG): container finished" podID="fc1665b7-e923-4048-b181-dc6383ccebb2" containerID="3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc" exitCode=0
Jul 02 17:32:32.612971 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.612703    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"
Jul 02 17:32:32.612971 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.612744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" event={"ID":"fc1665b7-e923-4048-b181-dc6383ccebb2","Type":"ContainerDied","Data":"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"}
Jul 02 17:32:32.612971 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.612787    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw" event={"ID":"fc1665b7-e923-4048-b181-dc6383ccebb2","Type":"ContainerDied","Data":"3fbf5f71b22db0a224d62fbe794669ce32736aa712dad63192a6bda46abe331f"}
Jul 02 17:32:32.612971 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.612804    2581 scope.go:117] "RemoveContainer" containerID="3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"
Jul 02 17:32:32.622148 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.622133    2581 scope.go:117] "RemoveContainer" containerID="3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"
Jul 02 17:32:32.622408 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:32:32.622389    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc\": container with ID starting with 3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc not found: ID does not exist" containerID="3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"
Jul 02 17:32:32.622479 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.622418    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc"} err="failed to get container status \"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc\": rpc error: code = NotFound desc = could not find container \"3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc\": container with ID starting with 3943d091276cf564b15c72651c7e10fc359df8d0700533d754e526252b3e2cfc not found: ID does not exist"
Jul 02 17:32:32.634379 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.634356    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:32:32.637950 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.637930    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8a6030/test-model-8a6030-kserve-84f57d4db9-kxvxw"]
Jul 02 17:32:32.815825 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:32.815795    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc1665b7-e923-4048-b181-dc6383ccebb2" path="/var/lib/kubelet/pods/fc1665b7-e923-4048-b181-dc6383ccebb2/volumes"
Jul 02 17:32:33.117122 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.117035    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" podUID="32e0f1d9-e00a-458d-9593-abefdf4c5702" containerName="main" containerID="cri-o://d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9" gracePeriod=45
Jul 02 17:32:33.367679 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.367611    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:33.464720 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.464720 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464721    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.465005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464748    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgv5d\" (UniqueName: \"kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.465005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464763    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.465005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.465005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464917    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache\") pod \"32e0f1d9-e00a-458d-9593-abefdf4c5702\" (UID: \"32e0f1d9-e00a-458d-9593-abefdf4c5702\") "
Jul 02 17:32:33.465005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.464976    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:33.465305 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.465149    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home" (OuterVolumeSpecName: "home") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:33.465305 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.465181    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache" (OuterVolumeSpecName: "model-cache") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:33.465305 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.465209    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.467078 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.467050    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d" (OuterVolumeSpecName: "kube-api-access-cgv5d") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "kube-api-access-cgv5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:32:33.467297 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.467266    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm" (OuterVolumeSpecName: "dshm") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:32:33.467381 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.467319    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "32e0f1d9-e00a-458d-9593-abefdf4c5702" (UID: "32e0f1d9-e00a-458d-9593-abefdf4c5702"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:32:33.566529 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.566495    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32e0f1d9-e00a-458d-9593-abefdf4c5702-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.566529 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.566525    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgv5d\" (UniqueName: \"kubernetes.io/projected/32e0f1d9-e00a-458d-9593-abefdf4c5702-kube-api-access-cgv5d\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.566529 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.566535    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.566750 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.566543    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.566750 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.566552    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32e0f1d9-e00a-458d-9593-abefdf4c5702-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:32:33.618755 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.618664    2581 generic.go:356] "Generic (PLEG): container finished" podID="32e0f1d9-e00a-458d-9593-abefdf4c5702" containerID="d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9" exitCode=0
Jul 02 17:32:33.618755 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.618712    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"
Jul 02 17:32:33.618987 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.618761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" event={"ID":"32e0f1d9-e00a-458d-9593-abefdf4c5702","Type":"ContainerDied","Data":"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"}
Jul 02 17:32:33.618987 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.618812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82" event={"ID":"32e0f1d9-e00a-458d-9593-abefdf4c5702","Type":"ContainerDied","Data":"1e9e5593285a95062b3245ef2b7efa1d7a2b0da46b5aff46973dd975d7a87844"}
Jul 02 17:32:33.618987 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.618839    2581 scope.go:117] "RemoveContainer" containerID="d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"
Jul 02 17:32:33.628030 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.628014    2581 scope.go:117] "RemoveContainer" containerID="d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"
Jul 02 17:32:33.628302 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:32:33.628283    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9\": container with ID starting with d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9 not found: ID does not exist" containerID="d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"
Jul 02 17:32:33.628363 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.628313    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9"} err="failed to get container status \"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9\": rpc error: code = NotFound desc = could not find container \"d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9\": container with ID starting with d3178417007aff8ab6462ab814fc380cc0db1c760b119590789adab8d69cecf9 not found: ID does not exist"
Jul 02 17:32:33.639722 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.639697    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:32:33.643687 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:33.643666    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-5569c1/test-model-5569c1-kserve-8f495dd79-djl82"]
Jul 02 17:32:34.816976 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:32:34.816943    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32e0f1d9-e00a-458d-9593-abefdf4c5702" path="/var/lib/kubelet/pods/32e0f1d9-e00a-458d-9593-abefdf4c5702/volumes"
Jul 02 17:33:05.116276 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:33:05.116238    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:33:19.934976 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:33:19.934943    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:33:35.608237 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:33:35.608203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:33:48.601537 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:33:48.601500    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:33:58.720431 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:33:58.720396    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:34:07.410969 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:07.410930    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:34:19.294749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.294717    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:34:19.295223 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295198    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32e0f1d9-e00a-458d-9593-abefdf4c5702" containerName="main"
Jul 02 17:34:19.295223 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295223    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="32e0f1d9-e00a-458d-9593-abefdf4c5702" containerName="main"
Jul 02 17:34:19.295317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295256    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc1665b7-e923-4048-b181-dc6383ccebb2" containerName="main"
Jul 02 17:34:19.295317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295265    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc1665b7-e923-4048-b181-dc6383ccebb2" containerName="main"
Jul 02 17:34:19.295384 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295331    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc1665b7-e923-4048-b181-dc6383ccebb2" containerName="main"
Jul 02 17:34:19.295384 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.295341    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="32e0f1d9-e00a-458d-9593-abefdf4c5702" containerName="main"
Jul 02 17:34:19.298438 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.298423    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.301039 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.301016    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"openshift-service-ca.crt\""
Jul 02 17:34:19.302317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.302296    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"auth-test-model-0e06fb-kserve-self-signed-certs\""
Jul 02 17:34:19.302317 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.302308    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"default-dockercfg-b67lq\""
Jul 02 17:34:19.302450 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.302353    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"kube-root-ca.crt\""
Jul 02 17:34:19.308388 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.308368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:34:19.346397 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.346559 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346407    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.346559 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346449    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.346559 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346518    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.346694 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.346694 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.346601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrwp\" (UniqueName: \"kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447237 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447204    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447435 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447261    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447435 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447435 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447317    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447435 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrwp\" (UniqueName: \"kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447435 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447752 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447682    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447752 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447723    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.447939 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.447917    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.450178 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.450153    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.450371 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.450354    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.455158 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.455134    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrwp\" (UniqueName: \"kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp\") pod \"auth-test-model-0e06fb-kserve-98d656ff-gdnjw\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") " pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.608418 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.608334    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:19.751349 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:19.751322    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:34:19.753446 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:34:19.753420    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37f61f98_4c33_48a7_b251_a5d0ea7ffa70.slice/crio-c67d2fd7adbee9290bc352ed03c8150f01b932b02ff82eb06c6639b32c38f892 WatchSource:0}: Error finding container c67d2fd7adbee9290bc352ed03c8150f01b932b02ff82eb06c6639b32c38f892: Status 404 returned error can't find the container with id c67d2fd7adbee9290bc352ed03c8150f01b932b02ff82eb06c6639b32c38f892
Jul 02 17:34:20.113903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:20.113869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" event={"ID":"37f61f98-4c33-48a7-b251-a5d0ea7ffa70","Type":"ContainerStarted","Data":"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"}
Jul 02 17:34:20.113903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:20.113911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" event={"ID":"37f61f98-4c33-48a7-b251-a5d0ea7ffa70","Type":"ContainerStarted","Data":"c67d2fd7adbee9290bc352ed03c8150f01b932b02ff82eb06c6639b32c38f892"}
Jul 02 17:34:20.129519 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:20.129472    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" podStartSLOduration=1.129458179 podStartE2EDuration="1.129458179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:34:20.129184102 +0000 UTC m=+2251.816185075" watchObservedRunningTime="2026-07-02 17:34:20.129458179 +0000 UTC m=+2251.816459166"
Jul 02 17:34:22.308322 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:22.308286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:34:29.609162 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:29.609121    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:29.609162 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:29.609172    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:29.622029 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:29.622004    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:30.173236 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:30.173206    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:34:40.195053 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.195018    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:34:40.198421 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.198398    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.201101 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.201082    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"openshift-service-ca.crt\""
Jul 02 17:34:40.201208 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.201089    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"kube-root-ca.crt\""
Jul 02 17:34:40.202309 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.202287    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"default-dockercfg-c27m7\""
Jul 02 17:34:40.202309 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.202302    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"auth-test-model-d87672-kserve-self-signed-certs\""
Jul 02 17:34:40.207989 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.207966    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:34:40.308535 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308503    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.308535 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308541    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.308736 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308569    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.308736 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308614    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn459\" (UniqueName: \"kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.308736 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308672    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.308736 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.308718    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409539    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409572    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409805 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409805 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409805 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409645    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn459\" (UniqueName: \"kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.409805 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.409704    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.410123 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.410090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.410212 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.410123    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.410212 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.410149    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.411821 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.411800    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.412171 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.412152    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.417575 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.417552    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn459\" (UniqueName: \"kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459\") pod \"auth-test-model-d87672-kserve-86d9bdb4c-8257q\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") " pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.509685 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.509595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:40.647112 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:40.647084    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:34:40.648738 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:34:40.648712    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42d4f642_f9bd_4bfc_94c7_98c33cdd8084.slice/crio-d531d827e22d10fb535dea7470d0e884863b1370e8f6bc52f95751bad919ac6d WatchSource:0}: Error finding container d531d827e22d10fb535dea7470d0e884863b1370e8f6bc52f95751bad919ac6d: Status 404 returned error can't find the container with id d531d827e22d10fb535dea7470d0e884863b1370e8f6bc52f95751bad919ac6d
Jul 02 17:34:41.216837 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:41.216803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" event={"ID":"42d4f642-f9bd-4bfc-94c7-98c33cdd8084","Type":"ContainerStarted","Data":"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"}
Jul 02 17:34:41.216837 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:41.216845    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" event={"ID":"42d4f642-f9bd-4bfc-94c7-98c33cdd8084","Type":"ContainerStarted","Data":"d531d827e22d10fb535dea7470d0e884863b1370e8f6bc52f95751bad919ac6d"}
Jul 02 17:34:41.238243 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:41.238193    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" podStartSLOduration=1.238176052 podStartE2EDuration="1.238176052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:34:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:34:41.235793803 +0000 UTC m=+2272.922794771" watchObservedRunningTime="2026-07-02 17:34:41.238176052 +0000 UTC m=+2272.925177024"
Jul 02 17:34:44.202181 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:44.202142    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:34:50.510281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:50.510227    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:50.510281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:50.510288    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:50.523086 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:50.523051    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:34:51.274747 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:34:51.274719    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:35:00.520738 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:35:00.520701    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:35:15.602679 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:35:15.602644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:35:31.805941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:35:31.805908    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:35:47.813925 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:35:47.813813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:36:05.205106 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:05.205066    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:36:22.914126 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:22.914090    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:36:39.105411 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:39.105371    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:36:43.419987 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:43.419943    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:36:44.280066 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:44.280032    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:36:48.949908 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:48.949882    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:36:48.953851 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:48.953829    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-x9mwx_51bec5ed-8842-47d8-8faa-9be7933368a9/ovn-acl-logging/0.log"
Jul 02 17:36:49.692436 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.692405    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:36:49.695751 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.695734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.698636 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.698613    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"sub-test-model-0e06fb-kserve-self-signed-certs\""
Jul 02 17:36:49.704592 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.704571    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:36:49.819499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819464    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.819499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819499    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vmcj\" (UniqueName: \"kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.819757 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.819757 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.819757 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819687    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.819757 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.819714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920385 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920385 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920381    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920635 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920411    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920635 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920527    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vmcj\" (UniqueName: \"kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920635 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920565    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920741 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.920774 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.920757    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.921120 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.921099    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.921208 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.921129    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.923052 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.923031    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.923128 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.923053    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:49.929054 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:49.929030    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vmcj\" (UniqueName: \"kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj\") pod \"sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") " pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:50.005396 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:50.005308    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:36:50.140040 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:50.140017    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:36:50.142298 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:36:50.142277    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b28c25c_53a6_493b_b713_c23ab8b110db.slice/crio-74100f2d3eb847aa04023dce7b545eec1e57d55961af53b7ed74d26232b40010 WatchSource:0}: Error finding container 74100f2d3eb847aa04023dce7b545eec1e57d55961af53b7ed74d26232b40010: Status 404 returned error can't find the container with id 74100f2d3eb847aa04023dce7b545eec1e57d55961af53b7ed74d26232b40010
Jul 02 17:36:50.832641 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:50.832605    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" event={"ID":"8b28c25c-53a6-493b-b713-c23ab8b110db","Type":"ContainerStarted","Data":"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"}
Jul 02 17:36:50.832827 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:50.832650    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" event={"ID":"8b28c25c-53a6-493b-b713-c23ab8b110db","Type":"ContainerStarted","Data":"74100f2d3eb847aa04023dce7b545eec1e57d55961af53b7ed74d26232b40010"}
Jul 02 17:36:50.851658 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:50.851606    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" podStartSLOduration=1.851589906 podStartE2EDuration="1.851589906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:36:50.85030485 +0000 UTC m=+2402.537305846" watchObservedRunningTime="2026-07-02 17:36:50.851589906 +0000 UTC m=+2402.538590877"
Jul 02 17:36:54.213038 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:54.213002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:36:58.443171 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.443131    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" podUID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" containerName="main" containerID="cri-o://5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6" gracePeriod=45
Jul 02 17:36:58.701320 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.701267    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:36:58.798490 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798458    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798501    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrrwp\" (UniqueName: \"kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798563    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798617    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798646 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir\") pod \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\" (UID: \"37f61f98-4c33-48a7-b251-a5d0ea7ffa70\") "
Jul 02 17:36:58.798941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798770    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home" (OuterVolumeSpecName: "home") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:58.799005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798920    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache" (OuterVolumeSpecName: "model-cache") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:58.799005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798957    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:58.799005 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.798968    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:58.800795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.800769    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:36:58.801049 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.801024    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm" (OuterVolumeSpecName: "dshm") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:58.801049 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.801043    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp" (OuterVolumeSpecName: "kube-api-access-xrrwp") pod "37f61f98-4c33-48a7-b251-a5d0ea7ffa70" (UID: "37f61f98-4c33-48a7-b251-a5d0ea7ffa70"). InnerVolumeSpecName "kube-api-access-xrrwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:36:58.872903 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.872850    2581 generic.go:356] "Generic (PLEG): container finished" podID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" containerID="5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6" exitCode=0
Jul 02 17:36:58.873059 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.872914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"
Jul 02 17:36:58.873059 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.872949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" event={"ID":"37f61f98-4c33-48a7-b251-a5d0ea7ffa70","Type":"ContainerDied","Data":"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"}
Jul 02 17:36:58.873059 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.873001    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw" event={"ID":"37f61f98-4c33-48a7-b251-a5d0ea7ffa70","Type":"ContainerDied","Data":"c67d2fd7adbee9290bc352ed03c8150f01b932b02ff82eb06c6639b32c38f892"}
Jul 02 17:36:58.873059 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.873024    2581 scope.go:117] "RemoveContainer" containerID="5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"
Jul 02 17:36:58.881694 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.881680    2581 scope.go:117] "RemoveContainer" containerID="5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"
Jul 02 17:36:58.881953 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:36:58.881933    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6\": container with ID starting with 5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6 not found: ID does not exist" containerID="5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"
Jul 02 17:36:58.882046 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.881960    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6"} err="failed to get container status \"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6\": rpc error: code = NotFound desc = could not find container \"5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6\": container with ID starting with 5b991ec61750895e29e7f18d489c90237d14bc531541be70913c1a9c72ecf4a6 not found: ID does not exist"
Jul 02 17:36:58.888679 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.888658    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:36:58.891408 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.891390    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/auth-test-model-0e06fb-kserve-98d656ff-gdnjw"]
Jul 02 17:36:58.899572 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.899550    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:58.899663 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.899574    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:58.899663 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.899586    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:58.899663 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.899598    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrwp\" (UniqueName: \"kubernetes.io/projected/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-kube-api-access-xrrwp\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:58.899663 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:58.899611    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f61f98-4c33-48a7-b251-a5d0ea7ffa70-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.300157 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.300122    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" podUID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" containerName="main" containerID="cri-o://cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef" gracePeriod=45
Jul 02 17:36:59.559991 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.559935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:36:59.706132 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706098    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706132 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706131    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706386 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706182    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706386 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706216    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706386 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706236    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706386 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706285    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wn459\" (UniqueName: \"kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459\") pod \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\" (UID: \"42d4f642-f9bd-4bfc-94c7-98c33cdd8084\") "
Jul 02 17:36:59.706588 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706418    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home" (OuterVolumeSpecName: "home") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:59.706588 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706496    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:59.706588 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706510    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache" (OuterVolumeSpecName: "model-cache") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:59.706588 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.706553    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.708601 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.708574    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm" (OuterVolumeSpecName: "dshm") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:36:59.708720 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.708606    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:36:59.708720 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.708654    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459" (OuterVolumeSpecName: "kube-api-access-wn459") pod "42d4f642-f9bd-4bfc-94c7-98c33cdd8084" (UID: "42d4f642-f9bd-4bfc-94c7-98c33cdd8084"). InnerVolumeSpecName "kube-api-access-wn459". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:36:59.807493 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.807461    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn459\" (UniqueName: \"kubernetes.io/projected/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-kube-api-access-wn459\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.807493 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.807489    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.807493 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.807499    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.807725 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.807509    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.807725 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.807517    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42d4f642-f9bd-4bfc-94c7-98c33cdd8084-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:36:59.879110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.879019    2581 generic.go:356] "Generic (PLEG): container finished" podID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" containerID="cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef" exitCode=0
Jul 02 17:36:59.879110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.879060    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"
Jul 02 17:36:59.879302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.879105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" event={"ID":"42d4f642-f9bd-4bfc-94c7-98c33cdd8084","Type":"ContainerDied","Data":"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"}
Jul 02 17:36:59.879302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.879150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q" event={"ID":"42d4f642-f9bd-4bfc-94c7-98c33cdd8084","Type":"ContainerDied","Data":"d531d827e22d10fb535dea7470d0e884863b1370e8f6bc52f95751bad919ac6d"}
Jul 02 17:36:59.879302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.879171    2581 scope.go:117] "RemoveContainer" containerID="cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"
Jul 02 17:36:59.888674 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.888655    2581 scope.go:117] "RemoveContainer" containerID="cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"
Jul 02 17:36:59.889057 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:36:59.889039    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef\": container with ID starting with cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef not found: ID does not exist" containerID="cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"
Jul 02 17:36:59.889130 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.889068    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef"} err="failed to get container status \"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef\": rpc error: code = NotFound desc = could not find container \"cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef\": container with ID starting with cd1506dafbf047f02c18ecd04bcc32b7b402fb1d3aa3caed42354350900fedef not found: ID does not exist"
Jul 02 17:36:59.900111 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.900089    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:36:59.903607 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:36:59.903588    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/auth-test-model-d87672-kserve-86d9bdb4c-8257q"]
Jul 02 17:37:00.006128 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.006099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:37:00.006303 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.006148    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:37:00.018293 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.018267    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:37:00.815798 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.815760    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" path="/var/lib/kubelet/pods/37f61f98-4c33-48a7-b251-a5d0ea7ffa70/volumes"
Jul 02 17:37:00.816152 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.816138    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" path="/var/lib/kubelet/pods/42d4f642-f9bd-4bfc-94c7-98c33cdd8084/volumes"
Jul 02 17:37:00.896943 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:00.896919    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:37:11.899161 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899078    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" containerName="main"
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899454    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" containerName="main"
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899470    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" containerName="main"
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899475    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" containerName="main"
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899535    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f61f98-4c33-48a7-b251-a5d0ea7ffa70" containerName="main"
Jul 02 17:37:11.899876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.899547    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="42d4f642-f9bd-4bfc-94c7-98c33cdd8084" containerName="main"
Jul 02 17:37:11.903639 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.903620    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:11.907633 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.907612    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"default-dockercfg-c27m7\""
Jul 02 17:37:11.907749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.907612    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"kube-root-ca.crt\""
Jul 02 17:37:11.907749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.907663    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"openshift-service-ca.crt\""
Jul 02 17:37:11.907749 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.907617    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"sub-test-model-d87672-kserve-self-signed-certs\""
Jul 02 17:37:11.913037 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:11.913015    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:37:12.012780 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.012746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.012970 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.012793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2gt\" (UniqueName: \"kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.012970 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.012901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.012970 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.012926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.013082 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.013009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.013082 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.013036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114011 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.113976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114011 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114010    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114049    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114281 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2gt\" (UniqueName: \"kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114484 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114462    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114550 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114491    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.114589 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.114566    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.116445 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.116414    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.116608 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.116592    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.123035 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.123015    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2gt\" (UniqueName: \"kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt\") pod \"sub-test-model-d87672-kserve-5649cf749f-lb6cf\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") " pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.213734 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.213643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:12.352049 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.351794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:37:12.354301 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:37:12.354273    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5179ffa_fbde_40a5_b0c7_bce4d5e14424.slice/crio-3d93022dfd559573742f506a8fcb1b5263c8e35e6c4e1d82f4f1bb6dc83d956d WatchSource:0}: Error finding container 3d93022dfd559573742f506a8fcb1b5263c8e35e6c4e1d82f4f1bb6dc83d956d: Status 404 returned error can't find the container with id 3d93022dfd559573742f506a8fcb1b5263c8e35e6c4e1d82f4f1bb6dc83d956d
Jul 02 17:37:12.940779 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.940746    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" event={"ID":"b5179ffa-fbde-40a5-b0c7-bce4d5e14424","Type":"ContainerStarted","Data":"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"}
Jul 02 17:37:12.940779 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.940786    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" event={"ID":"b5179ffa-fbde-40a5-b0c7-bce4d5e14424","Type":"ContainerStarted","Data":"3d93022dfd559573742f506a8fcb1b5263c8e35e6c4e1d82f4f1bb6dc83d956d"}
Jul 02 17:37:12.959707 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:12.959660    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" podStartSLOduration=1.95964617 podStartE2EDuration="1.95964617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:37:12.958000908 +0000 UTC m=+2424.645001888" watchObservedRunningTime="2026-07-02 17:37:12.95964617 +0000 UTC m=+2424.646647141"
Jul 02 17:37:16.009848 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:16.009813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:37:22.214294 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:22.214249    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:22.214294 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:22.214302    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:22.226664 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:22.226639    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:22.999919 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:22.999892    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:37:31.610147 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:31.610115    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:37:49.780179 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:49.780141    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:37:50.896247 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:50.896213    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:37:51.610523 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:51.610486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:37:56.621788 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.621755    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:37:56.625181 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.625162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.628491 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.628463    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-0e06fb\"/\"rate-test-model-0e06fb-kserve-self-signed-certs\""
Jul 02 17:37:56.640087 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.640059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:37:56.698197 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698164    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmdkq\" (UniqueName: \"kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.698351 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698214    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.698351 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.698351 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698286    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.698460 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.698460 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.698410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.798815 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.798781    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.798815 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.798821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmdkq\" (UniqueName: \"kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.798989    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799110 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799106    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799316 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799259    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799399 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799378    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.799440 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.799394    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.801302 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.801279    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.801499 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.801481    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.809796 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.809770    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmdkq\" (UniqueName: \"kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq\") pod \"rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") " pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:56.934438 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:56.934331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:37:57.079239 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:57.079205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:37:57.081696 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:37:57.081668    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9eabf62a_7440_4255_bada_42698966be6c.slice/crio-90113b71b8fcc1794a87c1a582b3a65c40651db3b4fd1d434b71fa4176c25dd6 WatchSource:0}: Error finding container 90113b71b8fcc1794a87c1a582b3a65c40651db3b4fd1d434b71fa4176c25dd6: Status 404 returned error can't find the container with id 90113b71b8fcc1794a87c1a582b3a65c40651db3b4fd1d434b71fa4176c25dd6
Jul 02 17:37:57.149392 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:57.149364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" event={"ID":"9eabf62a-7440-4255-bada-42698966be6c","Type":"ContainerStarted","Data":"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"}
Jul 02 17:37:57.149506 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:57.149403    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" event={"ID":"9eabf62a-7440-4255-bada-42698966be6c","Type":"ContainerStarted","Data":"90113b71b8fcc1794a87c1a582b3a65c40651db3b4fd1d434b71fa4176c25dd6"}
Jul 02 17:37:57.178297 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:37:57.178246    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" podStartSLOduration=1.178230231 podStartE2EDuration="1.178230231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:37:57.178204159 +0000 UTC m=+2468.865205134" watchObservedRunningTime="2026-07-02 17:37:57.178230231 +0000 UTC m=+2468.865231203"
Jul 02 17:38:04.800193 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:04.800158    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" podUID="8b28c25c-53a6-493b-b713-c23ab8b110db" containerName="main" containerID="cri-o://22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c" gracePeriod=45
Jul 02 17:38:05.057111 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.057056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:38:05.176341 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176314    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vmcj\" (UniqueName: \"kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176494 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176494 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176367    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176494 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176442    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176494 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176474    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176726 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176559    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home\") pod \"8b28c25c-53a6-493b-b713-c23ab8b110db\" (UID: \"8b28c25c-53a6-493b-b713-c23ab8b110db\") "
Jul 02 17:38:05.176795 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176707    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:05.176887 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176780    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache" (OuterVolumeSpecName: "model-cache") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:05.176887 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176826    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home" (OuterVolumeSpecName: "home") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:05.177006 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176921    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.177006 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.176941    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.178546 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.178522    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj" (OuterVolumeSpecName: "kube-api-access-6vmcj") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "kube-api-access-6vmcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:38:05.178546 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.178522    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm" (OuterVolumeSpecName: "dshm") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:05.178672 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.178524    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8b28c25c-53a6-493b-b713-c23ab8b110db" (UID: "8b28c25c-53a6-493b-b713-c23ab8b110db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:38:05.194278 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.194250    2581 generic.go:356] "Generic (PLEG): container finished" podID="8b28c25c-53a6-493b-b713-c23ab8b110db" containerID="22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c" exitCode=0
Jul 02 17:38:05.194375 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.194293    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"
Jul 02 17:38:05.194375 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.194327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" event={"ID":"8b28c25c-53a6-493b-b713-c23ab8b110db","Type":"ContainerDied","Data":"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"}
Jul 02 17:38:05.194375 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.194367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7" event={"ID":"8b28c25c-53a6-493b-b713-c23ab8b110db","Type":"ContainerDied","Data":"74100f2d3eb847aa04023dce7b545eec1e57d55961af53b7ed74d26232b40010"}
Jul 02 17:38:05.194481 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.194385    2581 scope.go:117] "RemoveContainer" containerID="22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"
Jul 02 17:38:05.202911 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.202895    2581 scope.go:117] "RemoveContainer" containerID="22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"
Jul 02 17:38:05.203154 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:38:05.203135    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c\": container with ID starting with 22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c not found: ID does not exist" containerID="22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"
Jul 02 17:38:05.203210 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.203163    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c"} err="failed to get container status \"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c\": rpc error: code = NotFound desc = could not find container \"22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c\": container with ID starting with 22b3c0003c38c3eef40a49476265a95a53bc423b5cb7de781ddde9426896be0c not found: ID does not exist"
Jul 02 17:38:05.217420 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.217399    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:38:05.221502 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.221482    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/sub-test-model-0e06fb-kserve-7c66c8499c-t7pz7"]
Jul 02 17:38:05.278229 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.278209    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.278229 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.278228    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vmcj\" (UniqueName: \"kubernetes.io/projected/8b28c25c-53a6-493b-b713-c23ab8b110db-kube-api-access-6vmcj\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.278365 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.278239    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8b28c25c-53a6-493b-b713-c23ab8b110db-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.278365 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.278249    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8b28c25c-53a6-493b-b713-c23ab8b110db-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:05.916610 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:05.916570    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" podUID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" containerName="main" containerID="cri-o://cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e" gracePeriod=45
Jul 02 17:38:06.169272 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.169206    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:38:06.208003 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.207970    2581 generic.go:356] "Generic (PLEG): container finished" podID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" containerID="cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e" exitCode=0
Jul 02 17:38:06.208140 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.208041    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"
Jul 02 17:38:06.208140 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.208058    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" event={"ID":"b5179ffa-fbde-40a5-b0c7-bce4d5e14424","Type":"ContainerDied","Data":"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"}
Jul 02 17:38:06.208140 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.208096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf" event={"ID":"b5179ffa-fbde-40a5-b0c7-bce4d5e14424","Type":"ContainerDied","Data":"3d93022dfd559573742f506a8fcb1b5263c8e35e6c4e1d82f4f1bb6dc83d956d"}
Jul 02 17:38:06.208140 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.208118    2581 scope.go:117] "RemoveContainer" containerID="cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"
Jul 02 17:38:06.218130 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.218112    2581 scope.go:117] "RemoveContainer" containerID="cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"
Jul 02 17:38:06.218381 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:38:06.218364    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e\": container with ID starting with cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e not found: ID does not exist" containerID="cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"
Jul 02 17:38:06.218432 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.218387    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e"} err="failed to get container status \"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e\": rpc error: code = NotFound desc = could not find container \"cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e\": container with ID starting with cc66ae14d5f2022c47c2f4795c9b00c2e7e152b90f98d5fd9d345ba9c4b0930e not found: ID does not exist"
Jul 02 17:38:06.287135 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287135 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287138    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287184    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287246    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn2gt\" (UniqueName: \"kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287315 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287264    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm\") pod \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\" (UID: \"b5179ffa-fbde-40a5-b0c7-bce4d5e14424\") "
Jul 02 17:38:06.287523 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287386    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:06.287523 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287430    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home" (OuterVolumeSpecName: "home") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:06.287628 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287526    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:06.287628 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287536    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.287628 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.287577    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.289367 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.289343    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:38:06.289482 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.289383    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm" (OuterVolumeSpecName: "dshm") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:38:06.289482 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.289396    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt" (OuterVolumeSpecName: "kube-api-access-jn2gt") pod "b5179ffa-fbde-40a5-b0c7-bce4d5e14424" (UID: "b5179ffa-fbde-40a5-b0c7-bce4d5e14424"). InnerVolumeSpecName "kube-api-access-jn2gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:38:06.388454 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.388428    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2gt\" (UniqueName: \"kubernetes.io/projected/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-kube-api-access-jn2gt\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.388454 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.388451    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.388615 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.388461    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.388615 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.388490    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5179ffa-fbde-40a5-b0c7-bce4d5e14424-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:38:06.532079 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.532049    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:38:06.538758 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.538738    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/sub-test-model-d87672-kserve-5649cf749f-lb6cf"]
Jul 02 17:38:06.815925 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.815898    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b28c25c-53a6-493b-b713-c23ab8b110db" path="/var/lib/kubelet/pods/8b28c25c-53a6-493b-b713-c23ab8b110db/volumes"
Jul 02 17:38:06.816278 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.816266    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" path="/var/lib/kubelet/pods/b5179ffa-fbde-40a5-b0c7-bce4d5e14424/volumes"
Jul 02 17:38:06.934449 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.934425    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:38:06.934823 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.934467    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:38:06.946320 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:06.946291    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:38:07.225432 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:07.225347    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:38:07.723377 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:07.723346    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:38:19.698133 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698104    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:38:19.698484 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698467    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b28c25c-53a6-493b-b713-c23ab8b110db" containerName="main"
Jul 02 17:38:19.698484 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698478    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b28c25c-53a6-493b-b713-c23ab8b110db" containerName="main"
Jul 02 17:38:19.698600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698494    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" containerName="main"
Jul 02 17:38:19.698600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698499    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" containerName="main"
Jul 02 17:38:19.698600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698566    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b28c25c-53a6-493b-b713-c23ab8b110db" containerName="main"
Jul 02 17:38:19.698600 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.698575    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5179ffa-fbde-40a5-b0c7-bce4d5e14424" containerName="main"
Jul 02 17:38:19.703033 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.703015    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.706613 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.706590    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"kube-root-ca.crt\""
Jul 02 17:38:19.707696 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.707679    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"default-dockercfg-c27m7\""
Jul 02 17:38:19.707963 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.707944    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"openshift-service-ca.crt\""
Jul 02 17:38:19.708043 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.707967    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-d87672\"/\"rate-test-model-d87672-kserve-self-signed-certs\""
Jul 02 17:38:19.713549 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.713525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:38:19.790493 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790468    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph788\" (UniqueName: \"kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.790619 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790534    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.790619 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790590    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.790619 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790615    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.790766 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790714    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.790766 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.790745    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.891941 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.891912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892074 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.891958    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph788\" (UniqueName: \"kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892074 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892014    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892074 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892074 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892051    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892259 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892115    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892429 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892399    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892531 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892421    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.892531 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.892461    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.894211 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.894184    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.894476 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.894460    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:19.909114 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:19.909090    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph788\" (UniqueName: \"kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788\") pod \"rate-test-model-d87672-kserve-b9c564ff-9m9v9\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") " pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:20.012424 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:20.012348    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:20.145034 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:20.145001    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:38:20.148297 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:38:20.148272    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda555884c_d541_491c_9fa1_c994b58bf4f7.slice/crio-c390604916407b58091ee37c9b5da03bd1849841359abbfb1a2a59ac22a75720 WatchSource:0}: Error finding container c390604916407b58091ee37c9b5da03bd1849841359abbfb1a2a59ac22a75720: Status 404 returned error can't find the container with id c390604916407b58091ee37c9b5da03bd1849841359abbfb1a2a59ac22a75720
Jul 02 17:38:20.274937 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:20.274837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" event={"ID":"a555884c-d541-491c-9fa1-c994b58bf4f7","Type":"ContainerStarted","Data":"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"}
Jul 02 17:38:20.274937 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:20.274911    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" event={"ID":"a555884c-d541-491c-9fa1-c994b58bf4f7","Type":"ContainerStarted","Data":"c390604916407b58091ee37c9b5da03bd1849841359abbfb1a2a59ac22a75720"}
Jul 02 17:38:20.294230 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:20.294184    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" podStartSLOduration=1.294171264 podStartE2EDuration="1.294171264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:38:20.293016553 +0000 UTC m=+2491.980017527" watchObservedRunningTime="2026-07-02 17:38:20.294171264 +0000 UTC m=+2491.981172236"
Jul 02 17:38:23.820234 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:23.820195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:38:30.012678 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:30.012644    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:30.013084 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:30.012691    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:30.025209 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:30.025188    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:30.335288 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:30.335264    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:38:37.440036 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:37.440004    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:38:57.615120 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:38:57.615077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:39:01.884543 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:01.884503    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:39:02.589985 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:02.589953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:39:07.520574 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:07.520536    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:39:16.904376 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:16.904314    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" podUID="9eabf62a-7440-4255-bada-42698966be6c" containerName="main" containerID="cri-o://1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289" gracePeriod=45
Jul 02 17:39:17.149943 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.149914    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:39:17.182129 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182067    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182129 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmdkq\" (UniqueName: \"kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182319 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182319 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182202    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182459 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182435    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182539 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182446    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.182539 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182521    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache\") pod \"9eabf62a-7440-4255-bada-42698966be6c\" (UID: \"9eabf62a-7440-4255-bada-42698966be6c\") "
Jul 02 17:39:17.182539 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182458    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home" (OuterVolumeSpecName: "home") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.182879 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182845    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.182988 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182883    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.182988 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.182889    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache" (OuterVolumeSpecName: "model-cache") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.184296 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.184273    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:39:17.184637 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.184621    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm" (OuterVolumeSpecName: "dshm") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.184728 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.184645    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq" (OuterVolumeSpecName: "kube-api-access-fmdkq") pod "9eabf62a-7440-4255-bada-42698966be6c" (UID: "9eabf62a-7440-4255-bada-42698966be6c"). InnerVolumeSpecName "kube-api-access-fmdkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:39:17.284167 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.284142    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9eabf62a-7440-4255-bada-42698966be6c-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.284167 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.284164    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmdkq\" (UniqueName: \"kubernetes.io/projected/9eabf62a-7440-4255-bada-42698966be6c-kube-api-access-fmdkq\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.284306 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.284173    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.284306 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.284181    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9eabf62a-7440-4255-bada-42698966be6c-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.546547 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.546519    2581 generic.go:356] "Generic (PLEG): container finished" podID="9eabf62a-7440-4255-bada-42698966be6c" containerID="1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289" exitCode=0
Jul 02 17:39:17.546691 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.546564    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"
Jul 02 17:39:17.546691 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.546611    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" event={"ID":"9eabf62a-7440-4255-bada-42698966be6c","Type":"ContainerDied","Data":"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"}
Jul 02 17:39:17.546691 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.546657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8" event={"ID":"9eabf62a-7440-4255-bada-42698966be6c","Type":"ContainerDied","Data":"90113b71b8fcc1794a87c1a582b3a65c40651db3b4fd1d434b71fa4176c25dd6"}
Jul 02 17:39:17.546691 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.546677    2581 scope.go:117] "RemoveContainer" containerID="1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"
Jul 02 17:39:17.557350 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.557324    2581 scope.go:117] "RemoveContainer" containerID="1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"
Jul 02 17:39:17.557716 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:39:17.557648    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289\": container with ID starting with 1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289 not found: ID does not exist" containerID="1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"
Jul 02 17:39:17.557716 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.557681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289"} err="failed to get container status \"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289\": rpc error: code = NotFound desc = could not find container \"1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289\": container with ID starting with 1d1f365f52b77689c057d20adaa7cc68d5fde35905948171b232bdb2f1bbd289 not found: ID does not exist"
Jul 02 17:39:17.579593 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.579567    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:39:17.581188 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.581169    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-0e06fb/rate-test-model-0e06fb-kserve-6dc4dcb695-vh4l8"]
Jul 02 17:39:17.609398 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.609342    2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" podUID="a555884c-d541-491c-9fa1-c994b58bf4f7" containerName="main" containerID="cri-o://dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92" gracePeriod=45
Jul 02 17:39:17.839338 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.839316    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:39:17.889541 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889516    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889681 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889570    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ph788\" (UniqueName: \"kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889681 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889612    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889681 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889639    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889844 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889844 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889712    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs\") pod \"a555884c-d541-491c-9fa1-c994b58bf4f7\" (UID: \"a555884c-d541-491c-9fa1-c994b58bf4f7\") "
Jul 02 17:39:17.889976 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889958    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home" (OuterVolumeSpecName: "home") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.890023 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.889972    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.890067 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.890029    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.891659 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.891636    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm" (OuterVolumeSpecName: "dshm") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 02 17:39:17.891805 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.891784    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788" (OuterVolumeSpecName: "kube-api-access-ph788") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "kube-api-access-ph788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 02 17:39:17.891852 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.891789    2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a555884c-d541-491c-9fa1-c994b58bf4f7" (UID: "a555884c-d541-491c-9fa1-c994b58bf4f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 02 17:39:17.991242 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991217    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-dshm\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.991242 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991239    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph788\" (UniqueName: \"kubernetes.io/projected/a555884c-d541-491c-9fa1-c994b58bf4f7-kube-api-access-ph788\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.991571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991253    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-home\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.991571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991264    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-model-cache\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.991571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991297    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a555884c-d541-491c-9fa1-c994b58bf4f7-tmp-dir\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:17.991571 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:17.991313    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a555884c-d541-491c-9fa1-c994b58bf4f7-tls-certs\") on node \"ip-10-0-139-28.ec2.internal\" DevicePath \"\""
Jul 02 17:39:18.553539 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.553507    2581 generic.go:356] "Generic (PLEG): container finished" podID="a555884c-d541-491c-9fa1-c994b58bf4f7" containerID="dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92" exitCode=0
Jul 02 17:39:18.553716 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.553552    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"
Jul 02 17:39:18.553716 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.553594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" event={"ID":"a555884c-d541-491c-9fa1-c994b58bf4f7","Type":"ContainerDied","Data":"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"}
Jul 02 17:39:18.553716 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.553637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9" event={"ID":"a555884c-d541-491c-9fa1-c994b58bf4f7","Type":"ContainerDied","Data":"c390604916407b58091ee37c9b5da03bd1849841359abbfb1a2a59ac22a75720"}
Jul 02 17:39:18.553716 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.553661    2581 scope.go:117] "RemoveContainer" containerID="dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"
Jul 02 17:39:18.562667 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.562649    2581 scope.go:117] "RemoveContainer" containerID="dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"
Jul 02 17:39:18.562969 ip-10-0-139-28 kubenswrapper[2581]: E0702 17:39:18.562944    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92\": container with ID starting with dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92 not found: ID does not exist" containerID="dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"
Jul 02 17:39:18.563034 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.562978    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92"} err="failed to get container status \"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92\": rpc error: code = NotFound desc = could not find container \"dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92\": container with ID starting with dc0193430d19c9c5cd2d80f8d551341fecbf96f01236322d4c94a33b44d50f92 not found: ID does not exist"
Jul 02 17:39:18.580572 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.580545    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:39:18.587983 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.587953    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-d87672/rate-test-model-d87672-kserve-b9c564ff-9m9v9"]
Jul 02 17:39:18.816977 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.816902    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9eabf62a-7440-4255-bada-42698966be6c" path="/var/lib/kubelet/pods/9eabf62a-7440-4255-bada-42698966be6c/volumes"
Jul 02 17:39:18.817283 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:39:18.817270    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a555884c-d541-491c-9fa1-c994b58bf4f7" path="/var/lib/kubelet/pods/a555884c-d541-491c-9fa1-c994b58bf4f7/volumes"
Jul 02 17:40:14.312475 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:14.312395    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:40:26.708360 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:26.708317    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-998h7"]
Jul 02 17:40:29.923408 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:29.923368    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c2mtt_f4b64ec2-c0cc-4d03-ab92-9631c302918a/manager/0.log"
Jul 02 17:40:30.045551 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:30.045522    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-th7ks_f9793e15-bd45-43b3-ad56-6aa147a2bedc/manager/0.log"
Jul 02 17:40:31.136252 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:31.136227    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7758cbfb47-kmr74_028ed0a1-722d-4ee3-840b-6ee93a2d2c7c/copy-manifests/0.log"
Jul 02 17:40:31.150742 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:31.150673    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7758cbfb47-kmr74_028ed0a1-722d-4ee3-840b-6ee93a2d2c7c/manager/0.log"
Jul 02 17:40:32.712046 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:32.712016    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5rzqk_29cd7b26-ce52-4976-bf50-45812e418bd4/manager/0.log"
Jul 02 17:40:32.828820 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:32.828793    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5lh94_79208780-cdda-401c-8174-f1202f030f74/manager/0.log"
Jul 02 17:40:32.943073 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:32.943051    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-25zsj_1203da23-9b52-49f7-a0d1-e227e9655f67/kuadrant-console-plugin/0.log"
Jul 02 17:40:33.057415 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:33.057392    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zhtj7_9e8a5974-2b52-4180-b1fd-a0f5f3d055d8/registry-server/0.log"
Jul 02 17:40:33.331736 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:33.331657    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-998h7_db3de8cc-7e73-4096-9b8d-f201abe807cb/limitador/0.log"
Jul 02 17:40:33.448038 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:33.448009    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nx9v6_dccf5c59-6b9e-4b6b-a99c-73abf094447a/manager/0.log"
Jul 02 17:40:34.046605 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:34.046581    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f48b874f-6v8wt_31cb3480-3219-4995-96b9-c01ee4243153/kube-auth-proxy/0.log"
Jul 02 17:40:34.281561 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:34.281532    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/1.log"
Jul 02 17:40:35.251154 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:35.251127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-h2j8z_a7607823-7780-46de-9382-b4de13c449cf/main/0.log"
Jul 02 17:40:35.364796 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:35.364767    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbhc4qq_19195264-789b-41e5-a242-ad9c4665afc7/main/0.log"
Jul 02 17:40:35.592064 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:35.592028    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-vx55j_e60974ae-4bbc-4f5c-89c0-9d5cafda1498/main/0.log"
Jul 02 17:40:39.698902 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.698842    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdk5z/must-gather-jbsl2"]
Jul 02 17:40:39.699269 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eabf62a-7440-4255-bada-42698966be6c" containerName="main"
Jul 02 17:40:39.699269 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699200    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eabf62a-7440-4255-bada-42698966be6c" containerName="main"
Jul 02 17:40:39.699269 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699228    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a555884c-d541-491c-9fa1-c994b58bf4f7" containerName="main"
Jul 02 17:40:39.699269 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699234    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a555884c-d541-491c-9fa1-c994b58bf4f7" containerName="main"
Jul 02 17:40:39.699402 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699286    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a555884c-d541-491c-9fa1-c994b58bf4f7" containerName="main"
Jul 02 17:40:39.699402 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.699298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eabf62a-7440-4255-bada-42698966be6c" containerName="main"
Jul 02 17:40:39.702511 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.702492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.705221 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.705200    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pdk5z\"/\"default-dockercfg-lz9qx\""
Jul 02 17:40:39.706531 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.706516    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdk5z\"/\"openshift-service-ca.crt\""
Jul 02 17:40:39.706596 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.706513    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pdk5z\"/\"kube-root-ca.crt\""
Jul 02 17:40:39.711770 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.711743    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdk5z/must-gather-jbsl2"]
Jul 02 17:40:39.754899 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.754853    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20171af0-31e1-4efc-8b22-bcd059455a2e-must-gather-output\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.755023 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.754901    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nh2fr\" (UniqueName: \"kubernetes.io/projected/20171af0-31e1-4efc-8b22-bcd059455a2e-kube-api-access-nh2fr\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.855555 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.855520    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20171af0-31e1-4efc-8b22-bcd059455a2e-must-gather-output\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.855555 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.855555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nh2fr\" (UniqueName: \"kubernetes.io/projected/20171af0-31e1-4efc-8b22-bcd059455a2e-kube-api-access-nh2fr\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.855925 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.855906    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/20171af0-31e1-4efc-8b22-bcd059455a2e-must-gather-output\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:39.864504 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:39.864477    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nh2fr\" (UniqueName: \"kubernetes.io/projected/20171af0-31e1-4efc-8b22-bcd059455a2e-kube-api-access-nh2fr\") pod \"must-gather-jbsl2\" (UID: \"20171af0-31e1-4efc-8b22-bcd059455a2e\") " pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:40.011553 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:40.011469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdk5z/must-gather-jbsl2"
Jul 02 17:40:40.143124 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:40.143095    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdk5z/must-gather-jbsl2"]
Jul 02 17:40:40.145594 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:40:40.145567    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20171af0_31e1_4efc_8b22_bcd059455a2e.slice/crio-5977b0d061aee4790b15a9428d5ce398062134816cdd3be3060409fcee83521f WatchSource:0}: Error finding container 5977b0d061aee4790b15a9428d5ce398062134816cdd3be3060409fcee83521f: Status 404 returned error can't find the container with id 5977b0d061aee4790b15a9428d5ce398062134816cdd3be3060409fcee83521f
Jul 02 17:40:40.147476 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:40.147459    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 02 17:40:40.935982 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:40.935949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdk5z/must-gather-jbsl2" event={"ID":"20171af0-31e1-4efc-8b22-bcd059455a2e","Type":"ContainerStarted","Data":"5977b0d061aee4790b15a9428d5ce398062134816cdd3be3060409fcee83521f"}
Jul 02 17:40:41.946894 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:41.946816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdk5z/must-gather-jbsl2" event={"ID":"20171af0-31e1-4efc-8b22-bcd059455a2e","Type":"ContainerStarted","Data":"164e1eefbde170f81f83112798a62a4f908aacfa4895443f00a1daa91109363a"}
Jul 02 17:40:41.947351 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:41.946908    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdk5z/must-gather-jbsl2" event={"ID":"20171af0-31e1-4efc-8b22-bcd059455a2e","Type":"ContainerStarted","Data":"e5c221af2ba632f1bcccdeb907ebb1e8eb09320a510fb8460fcd48a53376c682"}
Jul 02 17:40:41.964164 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:41.964116    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pdk5z/must-gather-jbsl2" podStartSLOduration=2.165913904 podStartE2EDuration="2.964099128s" totalImagesPullingTime="798.185224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 17:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:40:41.961709964 +0000 UTC m=+2633.648710936" watchObservedRunningTime="2026-07-02 17:40:41.964099128 +0000 UTC m=+2633.651100099"
Jul 02 17:40:42.495814 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:42.495783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hx8nl_a94f166e-7d6e-4716-9986-ea647fcb439c/global-pull-secret-syncer/0.log"
Jul 02 17:40:42.690978 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:42.690938    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wnmxk_25be8fe2-195c-4ff1-ab86-cefa434f6eae/konnectivity-agent/0.log"
Jul 02 17:40:42.733702 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:42.733668    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-28.ec2.internal_e757ec6aacc35d99d6a9719bfafce01a/haproxy/0.log"
Jul 02 17:40:47.745784 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:47.745749    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-5rzqk_29cd7b26-ce52-4976-bf50-45812e418bd4/manager/0.log"
Jul 02 17:40:47.774909 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:47.774883    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5lh94_79208780-cdda-401c-8174-f1202f030f74/manager/0.log"
Jul 02 17:40:47.803242 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:47.803202    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-25zsj_1203da23-9b52-49f7-a0d1-e227e9655f67/kuadrant-console-plugin/0.log"
Jul 02 17:40:47.848172 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:47.848136    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zhtj7_9e8a5974-2b52-4180-b1fd-a0f5f3d055d8/registry-server/0.log"
Jul 02 17:40:48.037267 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:48.037184    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-998h7_db3de8cc-7e73-4096-9b8d-f201abe807cb/limitador/0.log"
Jul 02 17:40:48.113449 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:48.113357    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-nx9v6_dccf5c59-6b9e-4b6b-a99c-73abf094447a/manager/0.log"
Jul 02 17:40:50.051012 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:50.050983    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdp2z_5eb8b86e-a545-4cb6-acd2-d1d89424721c/node-exporter/0.log"
Jul 02 17:40:50.073943 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:50.073914    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdp2z_5eb8b86e-a545-4cb6-acd2-d1d89424721c/kube-rbac-proxy/0.log"
Jul 02 17:40:50.096804 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:50.096773    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdp2z_5eb8b86e-a545-4cb6-acd2-d1d89424721c/init-textfile/0.log"
Jul 02 17:40:51.864606 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:51.864567    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cdb5968b9-c6vrg_16b7547d-4787-4c14-89ec-e781a2deee60/networking-console-plugin/0.log"
Jul 02 17:40:52.410879 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.410826    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"]
Jul 02 17:40:52.417497 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.417473    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.422724 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.422698    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"]
Jul 02 17:40:52.594819 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.594780    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-proc\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.595118 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.595099    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-sys\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.595259 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.595241    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-lib-modules\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.595415 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.595392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c8v7\" (UniqueName: \"kubernetes.io/projected/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-kube-api-access-8c8v7\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.595536 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.595470    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-podres\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696698 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696611    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c8v7\" (UniqueName: \"kubernetes.io/projected/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-kube-api-access-8c8v7\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696698 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-podres\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-proc\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-sys\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696813    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-lib-modules\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-podres\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696890    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-sys\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.696965 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696891    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-proc\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.697197 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.696980    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-lib-modules\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.705659 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.705620    2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c8v7\" (UniqueName: \"kubernetes.io/projected/252858eb-6cb4-4ec3-ba65-f0e0db8655a7-kube-api-access-8c8v7\") pod \"perf-node-gather-daemonset-zgvt8\" (UID: \"252858eb-6cb4-4ec3-ba65-f0e0db8655a7\") " pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.730554 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.730523    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:52.918318 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:52.918194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"]
Jul 02 17:40:52.927848 ip-10-0-139-28 kubenswrapper[2581]: W0702 17:40:52.927801    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod252858eb_6cb4_4ec3_ba65_f0e0db8655a7.slice/crio-602854bac49c8910ed89ca0ecfb97aa40bf3a57c1b51dcb79d32462b95f15887 WatchSource:0}: Error finding container 602854bac49c8910ed89ca0ecfb97aa40bf3a57c1b51dcb79d32462b95f15887: Status 404 returned error can't find the container with id 602854bac49c8910ed89ca0ecfb97aa40bf3a57c1b51dcb79d32462b95f15887
Jul 02 17:40:53.033880 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:53.031220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8" event={"ID":"252858eb-6cb4-4ec3-ba65-f0e0db8655a7","Type":"ContainerStarted","Data":"602854bac49c8910ed89ca0ecfb97aa40bf3a57c1b51dcb79d32462b95f15887"}
Jul 02 17:40:54.040567 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.040533    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8" event={"ID":"252858eb-6cb4-4ec3-ba65-f0e0db8655a7","Type":"ContainerStarted","Data":"787f2a814b83520cd7a4453c298ff61bcc7af37d92de079315c82c96a7ff84d0"}
Jul 02 17:40:54.041034 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.040602    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:40:54.059687 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.059631    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8" podStartSLOduration=2.059609452 podStartE2EDuration="2.059609452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 17:40:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 17:40:54.057174387 +0000 UTC m=+2645.744175359" watchObservedRunningTime="2026-07-02 17:40:54.059609452 +0000 UTC m=+2645.746610427"
Jul 02 17:40:54.707570 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.707541    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q28mw_dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8/dns/0.log"
Jul 02 17:40:54.728272 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.728236    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-q28mw_dbea20f0-9b8d-48a2-86ae-8a99d6c4dca8/kube-rbac-proxy/0.log"
Jul 02 17:40:54.840998 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:54.840975    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-k8bnn_9f23d5a2-8e77-4895-ab64-361cd53fbc90/dns-node-resolver/0.log"
Jul 02 17:40:55.375502 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:55.375464    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g5t5n_855cb017-d9f0-47da-800c-b325b3067bad/node-ca/0.log"
Jul 02 17:40:56.550712 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:56.550679    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f48b874f-6v8wt_31cb3480-3219-4995-96b9-c01ee4243153/kube-auth-proxy/0.log"
Jul 02 17:40:56.599241 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:56.599211    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/0.log"
Jul 02 17:40:56.643308 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:56.643281    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/1.log"
Jul 02 17:40:57.500113 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:57.500078    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vnppv_0b1f9e1e-f388-449b-bffc-8b3096117108/serve-healthcheck-canary/0.log"
Jul 02 17:40:57.930905 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:57.930878    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-bj6rc_54d70273-357a-456a-86aa-8f945e48f656/insights-operator/0.log"
Jul 02 17:40:57.931587 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:57.931568    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5f67b55555-bj6rc_54d70273-357a-456a-86aa-8f945e48f656/insights-operator/1.log"
Jul 02 17:40:58.084219 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:58.084170    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhb94_b9a39120-edd8-43ac-b7ff-016eee8f0658/kube-rbac-proxy/0.log"
Jul 02 17:40:58.103355 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:58.103324    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhb94_b9a39120-edd8-43ac-b7ff-016eee8f0658/exporter/0.log"
Jul 02 17:40:58.123876 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:40:58.123832    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dhb94_b9a39120-edd8-43ac-b7ff-016eee8f0658/extractor/0.log"
Jul 02 17:41:00.059238 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:00.059208    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pdk5z/perf-node-gather-daemonset-zgvt8"
Jul 02 17:41:00.239509 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:00.239483    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-c2mtt_f4b64ec2-c0cc-4d03-ab92-9631c302918a/manager/0.log"
Jul 02 17:41:00.404118 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:00.404033    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-th7ks_f9793e15-bd45-43b3-ad56-6aa147a2bedc/manager/0.log"
Jul 02 17:41:00.961162 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:00.961127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7758cbfb47-kmr74_028ed0a1-722d-4ee3-840b-6ee93a2d2c7c/manager/0.log"
Jul 02 17:41:00.979026 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:00.978998    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7758cbfb47-kmr74_028ed0a1-722d-4ee3-840b-6ee93a2d2c7c/copy-manifests/0.log"
Jul 02 17:41:03.046040 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:03.046010    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f48b874f-6v8wt_31cb3480-3219-4995-96b9-c01ee4243153/kube-auth-proxy/0.log"
Jul 02 17:41:03.085277 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:03.085244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/0.log"
Jul 02 17:41:03.131573 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:03.131527    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/1.log"
Jul 02 17:41:04.603022 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:04.602954    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54f48b874f-6v8wt_31cb3480-3219-4995-96b9-c01ee4243153/kube-auth-proxy/0.log"
Jul 02 17:41:04.638461 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:04.638433    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/0.log"
Jul 02 17:41:04.682082 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:04.682044    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-62d7s_2ea3470f-74ec-46fb-b2be-b88e9e0d4a47/istio-proxy/1.log"
Jul 02 17:41:07.472195 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:07.472167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-wxjkf_2cfd618b-02ee-455f-90da-44d33a7b58a5/migrator/0.log"
Jul 02 17:41:07.492283 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:07.492260    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-786f9fbc5-wxjkf_2cfd618b-02ee-455f-90da-44d33a7b58a5/graceful-termination/0.log"
Jul 02 17:41:08.924688 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:08.924666    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/kube-multus-additional-cni-plugins/0.log"
Jul 02 17:41:08.962744 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:08.962720    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/egress-router-binary-copy/0.log"
Jul 02 17:41:09.002912 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:09.002885    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/cni-plugins/0.log"
Jul 02 17:41:09.041725 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:09.041701    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/bond-cni-plugin/0.log"
Jul 02 17:41:09.082726 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:09.082708    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/routeoverride-cni/0.log"
Jul 02 17:41:09.136633 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:09.136607    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/whereabouts-cni-bincopy/0.log"
Jul 02 17:41:09.176663 ip-10-0-139-28 kubenswrapper[2581]: I0702 17:41:09.176605    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9s2wn_9b0f716e-c53d-401c-8f26-e65f68a94306/whereabouts-cni/0.log"