Jul 06 15:15:13.949685 ip-10-0-135-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 06 15:15:13.949696 ip-10-0-135-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 06 15:15:13.949706 ip-10-0-135-27 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 06 15:15:13.950013 ip-10-0-135-27 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 06 15:15:23.978017 ip-10-0-135-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 06 15:15:23.978036 ip-10-0-135-27 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 81889462dee84454ab227e6f709baadf --
Jul 06 15:16:59.101697 ip-10-0-135-27 systemd[1]: Starting Kubernetes Kubelet...
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.523261    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526932    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526946    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526949    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526952    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526956    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526959    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526967    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526970    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:16:59.593025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526972    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526975    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526978    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526980    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526983    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526986    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526988    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526991    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526993    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.526997    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527000    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527002    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527005    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527008    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527010    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527013    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527015    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527018    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527020    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527023    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:16:59.853976 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527026    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:16:59.595282 ip-10-0-135-27 systemd[1]: Started Kubernetes Kubelet.
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527029    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527031    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527034    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527036    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527039    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527041    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527044    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527046    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527049    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527052    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527054    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527056    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527059    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527061    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527064    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527066    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527068    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527071    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527074    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:16:59.863676 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527076    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527079    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527082    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527084    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527087    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527091    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527094    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527097    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527101    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527103    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527106    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527109    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527112    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527115    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527120    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527123    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527126    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527128    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527131    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:16:59.866729 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527134    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527137    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527139    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527142    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527145    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527147    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527150    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527152    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527154    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527157    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527159    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527162    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527164    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527166    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527169    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527172    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527175    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527178    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527180    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527184    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:16:59.869244 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527187    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527190    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527192    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527195    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527198    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527201    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527203    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527206    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527208    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527210    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527213    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527215    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527927    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527935    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527937    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527940    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527943    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527947    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527950    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:16:59.871249 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527953    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527957    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527961    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527964    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527967    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527970    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527972    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527975    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527978    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527980    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527983    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527985    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527989    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527992    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527994    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527997    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.527999    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528002    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528004    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528007    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:16:59.873515 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528010    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528012    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528016    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528019    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528022    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528025    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528027    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528030    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528032    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528035    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528037    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528040    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528043    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528046    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528049    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528051    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528055    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528059    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528063    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528065    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:16:59.875570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528068    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528071    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528073    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528076    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528079    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528082    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528084    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528087    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528089    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528092    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528095    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528097    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528100    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528102    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528105    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528108    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528110    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528113    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528115    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528118    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:16:59.878119 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528121    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528123    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528128    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528131    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528134    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528137    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528140    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528143    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528146    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528149    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528152    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528154    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528157    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528160    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528162    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528165    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528167    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528169    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528172    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:16:59.879915 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528174    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528177    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528179    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528182    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528184    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528190    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528192    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528195    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528197    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528200    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528202    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528205    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528207    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528599    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528605    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528607    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528611    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528614    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528617    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:16:59.882074 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528620    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528623    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528626    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528628    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528631    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528634    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528636    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528638    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528641    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528644    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528646    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528649    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528651    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528658    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528671    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528674    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528677    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528680    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528684    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528686    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:16:59.882876 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528689    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528691    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528694    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528696    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528699    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528701    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528703    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528706    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528710    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528714    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528717    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528720    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528723    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528725    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528727    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528730    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528732    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528735    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528737    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:16:59.883532 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528740    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528744    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528747    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528750    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528753    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528756    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528761    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528763    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528766    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528768    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528771    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528775    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528777    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528780    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528783    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528785    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528789    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528792    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528795    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:16:59.884128 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528798    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528800    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528803    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528805    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528808    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528810    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528814    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528816    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528819    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528821    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528824    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528826    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528828    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528831    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528834    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528836    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528839    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528841    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528843    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528848    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:16:59.884743 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528850    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528853    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528855    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528858    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528862    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528864    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528868    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528870    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528873    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528876    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528878    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528881    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528883    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528886    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.528889    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.528985    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.528995    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529001    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529006    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529010    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529014    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 06 15:16:59.885313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529018    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529022    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529025    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529028    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529031    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529035    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529038    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529041    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529044    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529047    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529051    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529054    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529061    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529064    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529067    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529072    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529075    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529079    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529082    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529085    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529088    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529091    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529094    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529097    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 06 15:16:59.886004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529101    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529103    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529108    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529110    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529114    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529116    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529120    2566 flags.go:64] FLAG: --enable-server="true"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529123    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529128    2566 flags.go:64] FLAG: --event-burst="100"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529131    2566 flags.go:64] FLAG: --event-qps="50"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529133    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529136    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529139    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529143    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529146    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529149    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529152    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529155    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529159    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529162    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529165    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529168    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529171    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529174    2566 flags.go:64] FLAG: --feature-gates=""
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529178    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 06 15:16:59.886641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529181    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529184    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529187    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529190    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529193    2566 flags.go:64] FLAG: --help="false"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529196    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-135-27.ec2.internal"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529199    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529202    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529205    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529208    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529211    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529214    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529217    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529219    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529222    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529225    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529229    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529232    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529235    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529238    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529241    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529243    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529246    2566 flags.go:64] FLAG: --lock-file=""
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529249    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 06 15:16:59.887343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529251    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529255    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529260    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529263    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529266    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529268    2566 flags.go:64] FLAG: --logging-format="text"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529272    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529276    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529278    2566 flags.go:64] FLAG: --manifest-url=""
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529281    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529285    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529288    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529292    2566 flags.go:64] FLAG: --max-pods="110"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529295    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529298    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529301    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529304    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529307    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529310    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529313    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529320    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529323    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529326    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 06 15:16:59.888001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529329    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529332    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529338    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529341    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529344    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529347    2566 flags.go:64] FLAG: --port="10250"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529350    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529353    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0075c8fa45585cc3e"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529356    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529358    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529361    2566 flags.go:64] FLAG: --register-node="true"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529367    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529370    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529373    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529376    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529380    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529383    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529386    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529402    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529405    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529408    2566 flags.go:64] FLAG: --runonce="false"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529411    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529414    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529417    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529420    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 06 15:16:59.888737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529423    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529426    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529429    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529432    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529434    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529437    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529440    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529443    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529446    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529449    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529452    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529457    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529460    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529463    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529467    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529470    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529473    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529476    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529480    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529483    2566 flags.go:64] FLAG: --v="2"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529486    2566 flags.go:64] FLAG: --version="false"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529490    2566 flags.go:64] FLAG: --vmodule=""
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529495    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.529498    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529614    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:16:59.889710 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529618    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529621    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529624    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529627    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529630    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529633    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529636    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529639    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529642    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529645    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529648    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529650    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529653    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529656    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529658    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529661    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529663    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529666    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529669    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:16:59.890405 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529672    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529674    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529677    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529680    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529683    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529685    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529689    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529691    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529694    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529696    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529700    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529703    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529706    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529708    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529711    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529714    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529716    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529719    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529722    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529724    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:17:00.027095 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529727    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529729    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529733    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529736    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529738    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529741    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529743    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529747    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529751    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529753    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529756    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529759    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529762    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529765    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529767    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529770    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529773    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529776    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529780    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:17:00.027870 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529782    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529785    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529788    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529791    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529794    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529797    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529799    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529802    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529805    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529808    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529810    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529813    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529815    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529818    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529820    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529823    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529825    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529828    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529830    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529833    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:17:00.028383 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529835    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529837    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529840    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529842    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529845    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529848    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529850    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529852    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529855    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529857    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529860    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529863    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529866    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529868    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529871    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529874    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529876    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529879    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529883    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:17:00.029025 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.529886    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.530406    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.537421    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.8"
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.537441    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.537467    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.537476    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537552    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537558    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537563    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537570    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537576    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537582    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537586    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537591    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:17:00.029557 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537595    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537600    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537604    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537608    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537612    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537616    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537620    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537624    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537630    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537634    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537638    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537644    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537648    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537652    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537656    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537659    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537663    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537667    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537671    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:17:00.030027 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537675    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537679    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537684    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537688    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537691    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537695    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537699    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537703    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537707    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537710    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537714    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537718    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537721    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537725    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537729    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537733    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537736    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537740    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537744    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537747    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:17:00.030563 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537751    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537754    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537758    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537761    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537765    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537769    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537773    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537776    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537780    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537784    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537788    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537792    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537796    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537801    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537806    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537809    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537812    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537814    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537817    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537820    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:17:00.031130 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537822    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537825    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537828    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537830    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537833    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537835    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537838    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537840    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537843    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537846    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537849    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537851    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537854    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537857    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537859    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537862    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537867    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537870    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537872    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537875    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:17:00.031711 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537878    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537880    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537883    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537885    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537888    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537890    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537893    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537896    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537898    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537901    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537903    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.537906    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.537911    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538054    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538061    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 06 15:17:00.032265 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538066    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538070    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538074    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538077    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538081    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538084    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538090    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538093    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538096    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538099    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538103    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538106    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538110    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538115    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538119    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538125    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538130    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538134    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538138    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 06 15:17:00.032716 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538142    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538146    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538149    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538153    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538157    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538160    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538164    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538169    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538172    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538176    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538179    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538183    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538187    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538191    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538194    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538198    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538202    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538205    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538209    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538215    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 06 15:17:00.033309 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538218    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538222    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538226    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538229    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538233    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538237    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538242    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538245    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538249    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538253    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538257    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538260    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538264    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538267    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538271    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538275    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538278    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538282    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538286    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538290    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 06 15:17:00.033848 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538294    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538298    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538303    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538307    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538310    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538314    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538318    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538322    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538326    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538330    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538333    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538338    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538342    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538346    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538349    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538353    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538357    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538361    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538366    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 06 15:17:00.034426 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538370    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538374    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538377    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538381    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538385    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538407    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538411    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538415    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538419    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538423    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538427    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538431    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538434    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538437    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538442    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538447    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538452    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538455    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 06 15:17:00.034946 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:16:59.538459    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.538467    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.538623    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.542741    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.543941    2566 server.go:1019] "Starting client certificate rotation"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.544047    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.544075    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.546520    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.550478    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.559953    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.561912    2566 log.go:25] "Validated CRI v1 image API"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.563610    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 06 15:17:00.035470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.566329    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7ced6eb4-8c29-414e-a633-32dae629e21f:/dev/nvme0n1p3 9cd32857-aeec-410f-9064-05d164aca221:/dev/nvme0n1p4]
Jul 06 15:17:00.035825 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.566343    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.572112    2566 manager.go:216] Machine: {Timestamp:2026-07-06 15:16:59.570678538 +0000 UTC m=+0.351107332 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100865 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24304177405090079cb38f683e908f SystemUUID:ec243041-7740-5090-079c-b38f683e908f BootID:81889462-dee8-4454-ab22-7e6f709baadf Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:40:84:76:e9:a3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:40:84:76:e9:a3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:1c:e3:d5:a7:95 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.572263    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.572360    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.575374    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.575461    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.576100    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.576111    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 06 15:17:00.035923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.576118    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.576855    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.578380    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.578526    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.581868    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.581983    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582012    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582025    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582048    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582060    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582071    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582506    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.582563    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.587906    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.589523    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.589550    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590448    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590461    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590467    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590473    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590482    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590487    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590495    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590500    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590507    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590512    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590521    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.590527    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.591173    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.591179    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.594358    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.594366    2566 server.go:1268] "Started kubelet"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.594431    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.594490    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.594569    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.595151    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.595627    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.596262    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.596379    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 06 15:17:00.036198 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.599221    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.602852    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.602918    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.603127    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603506    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603597    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603610    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603752    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603760    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603836    2566 factory.go:55] Registering systemd factory
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.603851    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.602700    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc0c05ca7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.594341543 +0000 UTC m=+0.374770335,LastTimestamp:2026-07-06 15:16:59.594341543 +0000 UTC m=+0.374770335,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604208    2566 factory.go:153] Registering CRI-O factory
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604227    2566 factory.go:223] Registration of the crio container factory successfully
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604286    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604312    2566 factory.go:103] Registering Raw factory
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604331    2566 manager.go:1196] Started watching for new ooms in manager
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.604901    2566 manager.go:349] Starting recovery of all containers
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.608157    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.608560    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.615954    2566 manager.go:354] Recovery completed
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.620416    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622065    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622093    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622111    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622669    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622680    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622696    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.622670    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.623422    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.623447    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.623486    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.624166    2566 policy_none.go:49] "None policy: Start"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.624177    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 06 15:17:00.037676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.624189    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.624713    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc2679c33  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622079539 +0000 UTC m=+0.402508331,LastTimestamp:2026-07-06 15:16:59.622079539 +0000 UTC m=+0.402508331,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.625126    2566 policy_none.go:47] "Start"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.634818    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc26800be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622105278 +0000 UTC m=+0.402534071,LastTimestamp:2026-07-06 15:16:59.622105278 +0000 UTC m=+0.402534071,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.637532    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc26826bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622115003 +0000 UTC m=+0.402543795,LastTimestamp:2026-07-06 15:16:59.622115003 +0000 UTC m=+0.402543795,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.645743    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-27.ec2.internal.18bfbc4dc2679c33\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc2679c33  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622079539 +0000 UTC m=+0.402508331,LastTimestamp:2026-07-06 15:16:59.623431861 +0000 UTC m=+0.403860658,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.041808 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.648978    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dzsq"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.650382    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-27.ec2.internal.18bfbc4dc26800be\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc26800be  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622105278 +0000 UTC m=+0.402534071,LastTimestamp:2026-07-06 15:16:59.623470053 +0000 UTC m=+0.403898846,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.657328    2566 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-27.ec2.internal.18bfbc4dc26826bb\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc26826bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-27.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.622115003 +0000 UTC m=+0.402543795,LastTimestamp:2026-07-06 15:16:59.6234931 +0000 UTC m=+0.403921891,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.658259    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4dzsq"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.703992    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707160    2566 manager.go:337] "Starting Device Plugin manager"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.707201    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707214    2566 server.go:87] "Starting device plugin registration server"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707517    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707528    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707655    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707727    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.707734    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.708271    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.708334    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.711618    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-27.ec2.internal.18bfbc4dc79ec863  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-27.ec2.internal,UID:ip-10-0-135-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-135-27.ec2.internal,},FirstTimestamp:2026-07-06 15:16:59.709581411 +0000 UTC m=+0.490010191,LastTimestamp:2026-07-06 15:16:59.709581411 +0000 UTC m=+0.490010191,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-27.ec2.internal,}"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.713673    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.714839    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.714869    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 06 15:17:00.042273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.714889    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.714924    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.727437    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.807876    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.808903    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.808932    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.808947    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.808976    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.814502    2566 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-27.ec2.internal\" not found" node="ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.815557    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"]
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.815635    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.818150    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.818175    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.818185    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.821315    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.821384    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.821420    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.821463    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.821459    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-27.ec2.internal\": node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823120    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823124    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823146    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823147    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823159    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.823164    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.824378    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.824418    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.825161    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.825194    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.825208    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.832253    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.858084    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-27.ec2.internal\" not found" node="ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.043640 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.863023    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-27.ec2.internal\" not found" node="ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.904898    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.904925    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:16:59.904943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef6bcc450bef4c891baf1f00f90df783-config\") pod \"kube-apiserver-proxy-ip-10-0-135-27.ec2.internal\" (UID: \"ef6bcc450bef4c891baf1f00f90df783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:16:59.932461    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef6bcc450bef4c891baf1f00f90df783-config\") pod \"kube-apiserver-proxy-ip-10-0-135-27.ec2.internal\" (UID: \"ef6bcc450bef4c891baf1f00f90df783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005801    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef6bcc450bef4c891baf1f00f90df783-config\") pod \"kube-apiserver-proxy-ip-10-0-135-27.ec2.internal\" (UID: \"ef6bcc450bef4c891baf1f00f90df783\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005829    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.005833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7a6bb8443d85847916080348a6754bf-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal\" (UID: \"f7a6bb8443d85847916080348a6754bf\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.044801 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.032833    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.145720 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.133744    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.241922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.160849    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.241922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.172577    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:00.241922 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.234443    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.410938 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.334950    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.485717 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.435429    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.564732 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.535973    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.564732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.536312    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:00.564732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.543459    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 06 15:17:00.564732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.543606    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 15:17:00.564732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.543629    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 06 15:17:00.638878 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.636709    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.661856    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 15:12:00 +0000 UTC" deadline="2028-01-02 10:36:39.715564585 +0000 UTC"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.661884    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13075h19m39.053684674s"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.679999    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.680973    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientMemory"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.681002    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasNoDiskPressure"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.681012    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeHasSufficientPID"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.688248    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:00.762015 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.737565    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.846270 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.838138    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.922204 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:00.922169    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6bcc450bef4c891baf1f00f90df783.slice/crio-0becef944197aa2ca83fd155b24019a29249293f41520ef574794f49513f0c43 WatchSource:0}: Error finding container 0becef944197aa2ca83fd155b24019a29249293f41520ef574794f49513f0c43: Status 404 returned error can't find the container with id 0becef944197aa2ca83fd155b24019a29249293f41520ef574794f49513f0c43
Jul 06 15:17:00.938460 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:00.938418    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:00.939370 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:00.939352    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:17:01.040618 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.040528    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-27.ec2.internal\" not found"
Jul 06 15:17:01.070137 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.070113    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:01.094168 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.094140    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7a6bb8443d85847916080348a6754bf.slice/crio-b2e0709b17a3495d2ae0bead6fa8ff49c0fc2ea7ad53034b294b50f1ae7ed388 WatchSource:0}: Error finding container b2e0709b17a3495d2ae0bead6fa8ff49c0fc2ea7ad53034b294b50f1ae7ed388: Status 404 returned error can't find the container with id b2e0709b17a3495d2ae0bead6fa8ff49c0fc2ea7ad53034b294b50f1ae7ed388
Jul 06 15:17:01.103879 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.103858    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:01.119687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.119663    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 15:17:01.120507 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.120493    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal"
Jul 06 15:17:01.130531 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.130515    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 06 15:17:01.389094 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.389067    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:01.583750 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.583718    2566 apiserver.go:52] "Watching apiserver"
Jul 06 15:17:01.592286 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.592258    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 06 15:17:01.592740 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.592716    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b","openshift-cluster-node-tuning-operator/tuned-fjdfn","openshift-image-registry/node-ca-9hhx2","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal","openshift-multus/multus-tpfz6","openshift-multus/network-metrics-daemon-7rrbh","kube-system/konnectivity-agent-74wlp","kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal","openshift-dns/node-resolver-d7dql","openshift-multus/multus-additional-cni-plugins-mbwwf","openshift-network-diagnostics/network-check-target-tg88d","openshift-network-operator/iptables-alerter-gb6tk","openshift-ovn-kubernetes/ovnkube-node-tsn7g","kube-system/global-pull-secret-syncer-92l2l"]
Jul 06 15:17:01.594534 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.594505    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.595681 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.595658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.596873 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.596853    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.597928 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.597900    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ts4hg\""
Jul 06 15:17:01.598049 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.597901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.598110 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.598082    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.599147 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.598787    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.599147 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.598811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 06 15:17:01.599147 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599034    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599210    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599327    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599720    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599730    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.599812    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fs6r9\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.600062    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.600153    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.600218    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.600287    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.600399    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.601001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.601083    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:01.602042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.601843    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-hswq6\""
Jul 06 15:17:01.602822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.602799    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.603815 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.603794    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 06 15:17:01.604084 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.604062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.604168 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.604130    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:01.605504 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.605483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-x6ccw\""
Jul 06 15:17:01.605598 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.605506    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 06 15:17:01.606034 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.606001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.607693 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.607560    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.608075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.608057    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.608871 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.608851    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-x62fk\""
Jul 06 15:17:01.609007 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.608913    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.609128 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.609100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.609919 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.609893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 06 15:17:01.610160 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.610139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rd9dd\""
Jul 06 15:17:01.610566 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.610544    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 06 15:17:01.610781 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.610763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.612294 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.612253    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.612294 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.612265    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.612657 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.612627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:01.612754 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.612707    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:01.613728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.613709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-9vhkj\""
Jul 06 15:17:01.613728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.613726    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kc4hf\""
Jul 06 15:17:01.613999 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.613980    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 06 15:17:01.614041 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.614022    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.614121 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.614099    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.614284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.614222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.615919 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.615887    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-k8s-cni-cncf-io\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.615928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc86a757-c267-46ce-8f42-b3428abd7517-hosts-file\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.616029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.615954    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-systemd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616000    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-bin\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-netd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616112    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6eb4b5c-48c6-461d-8350-a42e867d4715-serviceca\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616165    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-cni-binary-copy\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616193 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-os-release\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-socket-dir-parent\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616292    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-kubelet\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-slash\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-systemd-units\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616393    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-multus-certs\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-kubelet-config\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616464    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.616525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-multus\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-hostroot\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-script-lib\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-etc-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616703    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlfzs\" (UniqueName: \"kubernetes.io/projected/b57e3d86-626d-4705-b410-5694cfd1accd-kube-api-access-hlfzs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpkbb\" (UniqueName: \"kubernetes.io/projected/c6ab13cd-e191-471b-ada3-cf9055c0abc2-kube-api-access-qpkbb\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616752    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-log-socket\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616776    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-os-release\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-netns\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pn6\" (UniqueName: \"kubernetes.io/projected/0614eb8a-09a0-4524-9168-99ea0a851c7b-kube-api-access-97pn6\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-cnibin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.616988 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.616970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6eb4b5c-48c6-461d-8350-a42e867d4715-host\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617010    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvbvt\" (UniqueName: \"kubernetes.io/projected/fc86a757-c267-46ce-8f42-b3428abd7517-kube-api-access-lvbvt\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617037    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-multus-daemon-config\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-etc-kubernetes\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617088    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-var-lib-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617111    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-node-log\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617128    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-config\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617163    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58sh\" (UniqueName: \"kubernetes.io/projected/9a328533-20cf-48c0-bab3-da113fcaa421-kube-api-access-m58sh\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617202    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cnibin\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617242    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-env-overrides\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-conf-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617305    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-system-cni-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-netns\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-ovn\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.617752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617502    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-kubelet\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617603    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4jl77\""
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv9s8\" (UniqueName: \"kubernetes.io/projected/c6eb4b5c-48c6-461d-8350-a42e867d4715-kube-api-access-tv9s8\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-dbus\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617991    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-system-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617827    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.618021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-bin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.618046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc86a757-c267-46ce-8f42-b3428abd7517-tmp-dir\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.618074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.618508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.617901    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 06 15:17:01.619115 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.618703    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 06 15:17:01.650976 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.650912    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-999qr"
Jul 06 15:17:01.657791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.657761    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-999qr"
Jul 06 15:17:01.705420 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.705375    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlfzs\" (UniqueName: \"kubernetes.io/projected/b57e3d86-626d-4705-b410-5694cfd1accd-kube-api-access-hlfzs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpkbb\" (UniqueName: \"kubernetes.io/projected/c6ab13cd-e191-471b-ada3-cf9055c0abc2-kube-api-access-qpkbb\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-log-socket\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718431    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-sys-fs\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6cff18-a46a-424e-9f04-d3339ce97e70-host-slash\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718480    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-os-release\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718476    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-log-socket\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e6cff18-a46a-424e-9f04-d3339ce97e70-iptables-alerter-script\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-netns\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pn6\" (UniqueName: \"kubernetes.io/projected/0614eb8a-09a0-4524-9168-99ea0a851c7b-kube-api-access-97pn6\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-cnibin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6eb4b5c-48c6-461d-8350-a42e867d4715-host\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718674    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-netns\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvbvt\" (UniqueName: \"kubernetes.io/projected/fc86a757-c267-46ce-8f42-b3428abd7517-kube-api-access-lvbvt\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718722    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c6eb4b5c-48c6-461d-8350-a42e867d4715-host\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718622    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-os-release\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718786    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-conf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718789    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718822    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-cnibin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-registration-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmhcm\" (UniqueName: \"kubernetes.io/projected/7b72f750-9a65-424c-8930-54664350805b-kube-api-access-lmhcm\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-multus-daemon-config\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-etc-kubernetes\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-var-lib-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.719870 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.718982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-node-log\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-config\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719010    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-etc-kubernetes\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719026    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-var-lib-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58sh\" (UniqueName: \"kubernetes.io/projected/9a328533-20cf-48c0-bab3-da113fcaa421-kube-api-access-m58sh\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719056    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-node-log\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cnibin\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-env-overrides\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719135    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-conf-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719162    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-system-cni-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719184    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-binary-copy\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-netns\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719234    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cnibin\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719260    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-ovn\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719270    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-system-cni-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.720651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719266    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-netns\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-systemd\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719300    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-ovn\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719303    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-conf-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719346    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7g5ln\" (UniqueName: \"kubernetes.io/projected/4e6cff18-a46a-424e-9f04-d3339ce97e70-kube-api-access-7g5ln\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-kubelet\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv9s8\" (UniqueName: \"kubernetes.io/projected/c6eb4b5c-48c6-461d-8350-a42e867d4715-kube-api-access-tv9s8\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.719492    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.719572    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:02.219551257 +0000 UTC m=+2.999980053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719605    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-multus-daemon-config\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719617    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721384 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719496    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-modprobe-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719671    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-env-overrides\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-run\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719683    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-kubelet\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-lib-modules\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719717    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-config\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-tmp\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-dbus\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719857    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-dbus\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-system-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719892    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-bin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719943    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc86a757-c267-46ce-8f42-b3428abd7517-tmp-dir\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.721920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719987    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-system-cni-dir\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.719996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-k8s-cni-cncf-io\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720031    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc86a757-c267-46ce-8f42-b3428abd7517-hosts-file\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720063    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-bin\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-systemd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720123    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720127    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-k8s-cni-cncf-io\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720136    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fc86a757-c267-46ce-8f42-b3428abd7517-hosts-file\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720151    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-bin\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720178    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-run-systemd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720223    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-netd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720253    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-netd\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-cni-bin\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.722461 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720369    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720376    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6eb4b5c-48c6-461d-8350-a42e867d4715-serviceca\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-cni-binary-copy\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720448    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-os-release\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-socket-dir-parent\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720516    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc86a757-c267-46ce-8f42-b3428abd7517-tmp-dir\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720556    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720558    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-multus-socket-dir-parent\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720624    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-os-release\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720668    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-kubelet\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-slash\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.723177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720719    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysconfig\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-var-lib-kubelet\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720770    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c6ab13cd-e191-471b-ada3-cf9055c0abc2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-tuned\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720796    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-slash\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720748    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-host-kubelet\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720819    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mzqf\" (UniqueName: \"kubernetes.io/projected/d8533689-82cc-471a-a23d-17f5d0ab5af5-kube-api-access-7mzqf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-systemd-units\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-kubernetes\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720915    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-systemd-units\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-host\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720940    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c6eb4b5c-48c6-461d-8350-a42e867d4715-serviceca\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f9482-c85f-4d8c-952a-d3f575143f09-konnectivity-ca\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-device-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.724040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.720988    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9a328533-20cf-48c0-bab3-da113fcaa421-cni-binary-copy\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-multus-certs\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-kubelet-config\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721060    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-run-multus-certs\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721091    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-kubelet-config\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721102    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-multus\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721128    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-hostroot\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-script-lib\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721178    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-host-var-lib-cni-multus\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721180    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb6f9482-c85f-4d8c-952a-d3f575143f09-agent-certs\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.721198    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721232    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9a328533-20cf-48c0-bab3-da113fcaa421-hostroot\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-socket-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.724806 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.721290    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:02.221271229 +0000 UTC m=+3.001700018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721323    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-etc-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721380    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-sys\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721678    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovnkube-script-lib\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721736    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0614eb8a-09a0-4524-9168-99ea0a851c7b-etc-openvswitch\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.721939    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c6ab13cd-e191-471b-ada3-cf9055c0abc2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.725663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.724899    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0614eb8a-09a0-4524-9168-99ea0a851c7b-ovn-node-metrics-cert\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.726036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.725980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal" event={"ID":"f7a6bb8443d85847916080348a6754bf","Type":"ContainerStarted","Data":"b2e0709b17a3495d2ae0bead6fa8ff49c0fc2ea7ad53034b294b50f1ae7ed388"}
Jul 06 15:17:01.728383 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.728325    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pn6\" (UniqueName: \"kubernetes.io/projected/0614eb8a-09a0-4524-9168-99ea0a851c7b-kube-api-access-97pn6\") pod \"ovnkube-node-tsn7g\" (UID: \"0614eb8a-09a0-4524-9168-99ea0a851c7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.728785 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.728707    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal" event={"ID":"ef6bcc450bef4c891baf1f00f90df783","Type":"ContainerStarted","Data":"0becef944197aa2ca83fd155b24019a29249293f41520ef574794f49513f0c43"}
Jul 06 15:17:01.730457 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.730402    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58sh\" (UniqueName: \"kubernetes.io/projected/9a328533-20cf-48c0-bab3-da113fcaa421-kube-api-access-m58sh\") pod \"multus-tpfz6\" (UID: \"9a328533-20cf-48c0-bab3-da113fcaa421\") " pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.732171 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.731693    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpkbb\" (UniqueName: \"kubernetes.io/projected/c6ab13cd-e191-471b-ada3-cf9055c0abc2-kube-api-access-qpkbb\") pod \"multus-additional-cni-plugins-mbwwf\" (UID: \"c6ab13cd-e191-471b-ada3-cf9055c0abc2\") " pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.732171 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.732147    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv9s8\" (UniqueName: \"kubernetes.io/projected/c6eb4b5c-48c6-461d-8350-a42e867d4715-kube-api-access-tv9s8\") pod \"node-ca-9hhx2\" (UID: \"c6eb4b5c-48c6-461d-8350-a42e867d4715\") " pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.732171 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.732142    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvbvt\" (UniqueName: \"kubernetes.io/projected/fc86a757-c267-46ce-8f42-b3428abd7517-kube-api-access-lvbvt\") pod \"node-resolver-d7dql\" (UID: \"fc86a757-c267-46ce-8f42-b3428abd7517\") " pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.732979 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.732937    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlfzs\" (UniqueName: \"kubernetes.io/projected/b57e3d86-626d-4705-b410-5694cfd1accd-kube-api-access-hlfzs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:01.822422 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-systemd\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822478    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7g5ln\" (UniqueName: \"kubernetes.io/projected/4e6cff18-a46a-424e-9f04-d3339ce97e70-kube-api-access-7g5ln\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822524    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-modprobe-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822553    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-run\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-lib-modules\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-tmp\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822625    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysconfig\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-var-lib-kubelet\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822678    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-tuned\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822690    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-modprobe-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mzqf\" (UniqueName: \"kubernetes.io/projected/d8533689-82cc-471a-a23d-17f5d0ab5af5-kube-api-access-7mzqf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-kubernetes\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-host\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f9482-c85f-4d8c-952a-d3f575143f09-konnectivity-ca\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822858    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-etc-selinux\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-device-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822902    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-var-lib-kubelet\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.822933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb6f9482-c85f-4d8c-952a-d3f575143f09-agent-certs\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.823539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822960    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysconfig\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.823539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-socket-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.823539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.823480    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-socket-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.823539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.823530    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-host\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.823776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.823750    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-kubernetes\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.824513 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.824479    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/fb6f9482-c85f-4d8c-952a-d3f575143f09-konnectivity-ca\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.826178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.825952    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-device-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826094    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-lib-modules\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.822523    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-systemd\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826105    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-tuned\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826153    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8533689-82cc-471a-a23d-17f5d0ab5af5-tmp\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826203    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-run\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826312    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/fb6f9482-c85f-4d8c-952a-d3f575143f09-agent-certs\") pod \"konnectivity-agent-74wlp\" (UID: \"fb6f9482-c85f-4d8c-952a-d3f575143f09\") " pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826318    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-sys\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826389    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-sys\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826428    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-sys-fs\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826505    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-sys-fs\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826520    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6cff18-a46a-424e-9f04-d3339ce97e70-host-slash\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826583    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e6cff18-a46a-424e-9f04-d3339ce97e70-iptables-alerter-script\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4e6cff18-a46a-424e-9f04-d3339ce97e70-host-slash\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826626    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-conf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-registration-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmhcm\" (UniqueName: \"kubernetes.io/projected/7b72f750-9a65-424c-8930-54664350805b-kube-api-access-lmhcm\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826842    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-d\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.826984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.826982    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d8533689-82cc-471a-a23d-17f5d0ab5af5-etc-sysctl-conf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.827362 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.827040    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b72f750-9a65-424c-8930-54664350805b-registration-dir\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.827362 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.827196    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4e6cff18-a46a-424e-9f04-d3339ce97e70-iptables-alerter-script\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.831661 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.831638    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:01.831772 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.831669    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:01.831772 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.831685    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:01.831772 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:01.831756    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:02.331737734 +0000 UTC m=+3.112166550 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:01.835137 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.835109    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mzqf\" (UniqueName: \"kubernetes.io/projected/d8533689-82cc-471a-a23d-17f5d0ab5af5-kube-api-access-7mzqf\") pod \"tuned-fjdfn\" (UID: \"d8533689-82cc-471a-a23d-17f5d0ab5af5\") " pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.835710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.835689    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7g5ln\" (UniqueName: \"kubernetes.io/projected/4e6cff18-a46a-424e-9f04-d3339ce97e70-kube-api-access-7g5ln\") pod \"iptables-alerter-gb6tk\" (UID: \"4e6cff18-a46a-424e-9f04-d3339ce97e70\") " pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.841886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.841863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmhcm\" (UniqueName: \"kubernetes.io/projected/7b72f750-9a65-424c-8930-54664350805b-kube-api-access-lmhcm\") pod \"aws-ebs-csi-driver-node-c925b\" (UID: \"7b72f750-9a65-424c-8930-54664350805b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.909680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.909587    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mbwwf"
Jul 06 15:17:01.919725 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.919338    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:01.924616 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.924581    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6ab13cd_e191_471b_ada3_cf9055c0abc2.slice/crio-59ea041b445b8b2521a11daa3abbafddaccd8ae01672d8ec2df2b05be3493093 WatchSource:0}: Error finding container 59ea041b445b8b2521a11daa3abbafddaccd8ae01672d8ec2df2b05be3493093: Status 404 returned error can't find the container with id 59ea041b445b8b2521a11daa3abbafddaccd8ae01672d8ec2df2b05be3493093
Jul 06 15:17:01.927180 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.927152    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hhx2"
Jul 06 15:17:01.932626 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.932382    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0614eb8a_09a0_4524_9168_99ea0a851c7b.slice/crio-5e847322ec11b648489daf46b2370d20bf3a479a8be273eb9831c7cda8be22d4 WatchSource:0}: Error finding container 5e847322ec11b648489daf46b2370d20bf3a479a8be273eb9831c7cda8be22d4: Status 404 returned error can't find the container with id 5e847322ec11b648489daf46b2370d20bf3a479a8be273eb9831c7cda8be22d4
Jul 06 15:17:01.933822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.933751    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tpfz6"
Jul 06 15:17:01.939977 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.939939    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6eb4b5c_48c6_461d_8350_a42e867d4715.slice/crio-e51360fe71dd869b36904202ca5a16216c74c572acc12b3f83e8f9563e393be4 WatchSource:0}: Error finding container e51360fe71dd869b36904202ca5a16216c74c572acc12b3f83e8f9563e393be4: Status 404 returned error can't find the container with id e51360fe71dd869b36904202ca5a16216c74c572acc12b3f83e8f9563e393be4
Jul 06 15:17:01.942143 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.941966    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-d7dql"
Jul 06 15:17:01.945887 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.945864    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a328533_20cf_48c0_bab3_da113fcaa421.slice/crio-29193b5d31839cb88766b23e64095658064dc67d2908acf4c03d8ae8f72de2bc WatchSource:0}: Error finding container 29193b5d31839cb88766b23e64095658064dc67d2908acf4c03d8ae8f72de2bc: Status 404 returned error can't find the container with id 29193b5d31839cb88766b23e64095658064dc67d2908acf4c03d8ae8f72de2bc
Jul 06 15:17:01.949179 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.949155    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:01.952658 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.952633    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc86a757_c267_46ce_8f42_b3428abd7517.slice/crio-d98c39979baf14329c9c8e1a58d5b20b5887486da05260a7162526048760377e WatchSource:0}: Error finding container d98c39979baf14329c9c8e1a58d5b20b5887486da05260a7162526048760377e: Status 404 returned error can't find the container with id d98c39979baf14329c9c8e1a58d5b20b5887486da05260a7162526048760377e
Jul 06 15:17:01.958376 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.958354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fjdfn"
Jul 06 15:17:01.964730 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.964699    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb6f9482_c85f_4d8c_952a_d3f575143f09.slice/crio-b5a9062365eb447f921d265f164a6f676b4374a70d0554de64bf476acf8e2296 WatchSource:0}: Error finding container b5a9062365eb447f921d265f164a6f676b4374a70d0554de64bf476acf8e2296: Status 404 returned error can't find the container with id b5a9062365eb447f921d265f164a6f676b4374a70d0554de64bf476acf8e2296
Jul 06 15:17:01.967251 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.967033    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b"
Jul 06 15:17:01.970922 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.970832    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8533689_82cc_471a_a23d_17f5d0ab5af5.slice/crio-3089ffbd6e25e502ede382d925700e81f8ca6a71266ad1f29cc2616f82f355df WatchSource:0}: Error finding container 3089ffbd6e25e502ede382d925700e81f8ca6a71266ad1f29cc2616f82f355df: Status 404 returned error can't find the container with id 3089ffbd6e25e502ede382d925700e81f8ca6a71266ad1f29cc2616f82f355df
Jul 06 15:17:01.977577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:01.977327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gb6tk"
Jul 06 15:17:01.979225 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.979191    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b72f750_9a65_424c_8930_54664350805b.slice/crio-cc9c9c2754910f052bb7bd964e49aadd88fe2907ac445ffd3762ffd318776a0e WatchSource:0}: Error finding container cc9c9c2754910f052bb7bd964e49aadd88fe2907ac445ffd3762ffd318776a0e: Status 404 returned error can't find the container with id cc9c9c2754910f052bb7bd964e49aadd88fe2907ac445ffd3762ffd318776a0e
Jul 06 15:17:01.989795 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:01.989765    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e6cff18_a46a_424e_9f04_d3339ce97e70.slice/crio-35a81ec89b71c4346571c716a7cf3c4c0998e0ae45a609d2ab1c9d0769528f76 WatchSource:0}: Error finding container 35a81ec89b71c4346571c716a7cf3c4c0998e0ae45a609d2ab1c9d0769528f76: Status 404 returned error can't find the container with id 35a81ec89b71c4346571c716a7cf3c4c0998e0ae45a609d2ab1c9d0769528f76
Jul 06 15:17:02.229879 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.229782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:02.229879 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.229849    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:02.230109 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.229948    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:02.230109 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.229949    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:02.230109 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.230015    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:03.229994017 +0000 UTC m=+4.010422798 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:02.230109 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.230036    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:03.230026445 +0000 UTC m=+4.010455230 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:02.431579 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.431543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:02.431743 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.431720    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:02.431783 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.431750    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:02.431783 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.431761    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:02.431851 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.431818    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:03.431801084 +0000 UTC m=+4.212229892 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:02.660020 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.659958    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:12:02 +0000 UTC" deadline="2027-11-29 19:05:06.225332001 +0000 UTC"
Jul 06 15:17:02.660020 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.660015    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12267h48m3.565320839s"
Jul 06 15:17:02.715811 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.715767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:02.715961 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.715777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:02.715961 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.715903    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:02.716080 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.716017    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:02.716080 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.715777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:02.716189 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:02.716168    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:02.732232 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.732194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gb6tk" event={"ID":"4e6cff18-a46a-424e-9f04-d3339ce97e70","Type":"ContainerStarted","Data":"35a81ec89b71c4346571c716a7cf3c4c0998e0ae45a609d2ab1c9d0769528f76"}
Jul 06 15:17:02.733565 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.733537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjdfn" event={"ID":"d8533689-82cc-471a-a23d-17f5d0ab5af5","Type":"ContainerStarted","Data":"3089ffbd6e25e502ede382d925700e81f8ca6a71266ad1f29cc2616f82f355df"}
Jul 06 15:17:02.734924 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.734893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"5e847322ec11b648489daf46b2370d20bf3a479a8be273eb9831c7cda8be22d4"}
Jul 06 15:17:02.736118 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.736088    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerStarted","Data":"59ea041b445b8b2521a11daa3abbafddaccd8ae01672d8ec2df2b05be3493093"}
Jul 06 15:17:02.737372 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.737343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b" event={"ID":"7b72f750-9a65-424c-8930-54664350805b","Type":"ContainerStarted","Data":"cc9c9c2754910f052bb7bd964e49aadd88fe2907ac445ffd3762ffd318776a0e"}
Jul 06 15:17:02.738627 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.738601    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-74wlp" event={"ID":"fb6f9482-c85f-4d8c-952a-d3f575143f09","Type":"ContainerStarted","Data":"b5a9062365eb447f921d265f164a6f676b4374a70d0554de64bf476acf8e2296"}
Jul 06 15:17:02.739777 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.739755    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7dql" event={"ID":"fc86a757-c267-46ce-8f42-b3428abd7517","Type":"ContainerStarted","Data":"d98c39979baf14329c9c8e1a58d5b20b5887486da05260a7162526048760377e"}
Jul 06 15:17:02.740903 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.740878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tpfz6" event={"ID":"9a328533-20cf-48c0-bab3-da113fcaa421","Type":"ContainerStarted","Data":"29193b5d31839cb88766b23e64095658064dc67d2908acf4c03d8ae8f72de2bc"}
Jul 06 15:17:02.741909 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:02.741888    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hhx2" event={"ID":"c6eb4b5c-48c6-461d-8350-a42e867d4715","Type":"ContainerStarted","Data":"e51360fe71dd869b36904202ca5a16216c74c572acc12b3f83e8f9563e393be4"}
Jul 06 15:17:03.238503 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.238460    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:03.238719 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.238554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:03.238719 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.238713    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:03.238859 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.238780    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:05.238761862 +0000 UTC m=+6.019190667 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:03.239234 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.239211    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:03.239308 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.239292    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:05.239256212 +0000 UTC m=+6.019685009 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:03.441547 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.440845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:03.441547 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.441034    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:03.441547 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.441053    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:03.441547 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.441068    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:03.441547 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:03.441128    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:05.441108796 +0000 UTC m=+6.221537598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:03.573452 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.573163    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 06 15:17:03.660436 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.660138    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 15:12:02 +0000 UTC" deadline="2028-02-04 13:59:51.498256379 +0000 UTC"
Jul 06 15:17:03.660436 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.660429    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13870h42m47.837850294s"
Jul 06 15:17:03.747721 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.747676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal" event={"ID":"ef6bcc450bef4c891baf1f00f90df783","Type":"ContainerStarted","Data":"3df9c352f50acb0fa37545b9c2f6024fec60814c20705a1738952462b1ea93a3"}
Jul 06 15:17:03.763904 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:03.763822    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-27.ec2.internal" podStartSLOduration=2.763800684 podStartE2EDuration="2.763800684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:03.763436634 +0000 UTC m=+4.543865437" watchObservedRunningTime="2026-07-06 15:17:03.763800684 +0000 UTC m=+4.544229493"
Jul 06 15:17:04.715694 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:04.715662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:04.715694 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:04.715691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:04.716225 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:04.715810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:04.716225 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:04.716000    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:04.716225 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:04.716122    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:04.716225 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:04.716208    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:04.785397 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:04.785308    2566 generic.go:356] "Generic (PLEG): container finished" podID="f7a6bb8443d85847916080348a6754bf" containerID="394a5ff60b496ce9ed6900582af6c3785b6fe14f064388682946793e0238100d" exitCode=0
Jul 06 15:17:04.786121 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:04.785997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal" event={"ID":"f7a6bb8443d85847916080348a6754bf","Type":"ContainerDied","Data":"394a5ff60b496ce9ed6900582af6c3785b6fe14f064388682946793e0238100d"}
Jul 06 15:17:05.258492 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:05.258442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:05.258748 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:05.258525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:05.258748 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.258670    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:05.258748 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.258741    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:09.258717366 +0000 UTC m=+10.039146166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:05.259195 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.259174    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:05.259270 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.259236    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:09.259219899 +0000 UTC m=+10.039648698 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:05.460719 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:05.460672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:05.460935 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.460870    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:05.460935 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.460892    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:05.460935 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.460905    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:05.461093 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:05.460966    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:09.460947153 +0000 UTC m=+10.241375938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:05.798973 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:05.798883    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal" event={"ID":"f7a6bb8443d85847916080348a6754bf","Type":"ContainerStarted","Data":"595df0dd76d94b1c1eb72301274d37c92953cc6f34f01d839cd2d28c4300425e"}
Jul 06 15:17:05.817411 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:05.817338    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-27.ec2.internal" podStartSLOduration=4.8173126790000005 podStartE2EDuration="4.817312679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:05.816648047 +0000 UTC m=+6.597076853" watchObservedRunningTime="2026-07-06 15:17:05.817312679 +0000 UTC m=+6.597741485"
Jul 06 15:17:06.715631 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:06.715592    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:06.715836 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:06.715640    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:06.715836 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:06.715732    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:06.715951 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:06.715827    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:06.715951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:06.715878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:06.716065 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:06.715951    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:08.715866 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:08.715814    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:08.715866 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:08.715845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:08.716480 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:08.715845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:08.716480 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:08.715965    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:08.716480 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:08.716109    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:08.716480 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:08.716203    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:09.292101    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:09.292184    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.292324    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.292393    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:17.292370718 +0000 UTC m=+18.072799521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.292523    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:09.292622 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.292580    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:17.292563704 +0000 UTC m=+18.072992495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:09.493583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:09.493534    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:09.493827 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.493775    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:09.493827 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.493808    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:09.493827 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.493824    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:09.494055 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:09.493895    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:17.4938748 +0000 UTC m=+18.274303587 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:10.715680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:10.715641    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:10.715680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:10.715669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:10.716196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:10.715657    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:10.716196 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:10.715761    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:10.716196 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:10.715810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:10.716196 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:10.715905    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:12.715252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:12.715217    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:12.715765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:12.715225    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:12.715765 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:12.715354    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:12.715765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:12.715365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:12.715765 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:12.715480    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:12.715765 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:12.715561    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:14.715466 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:14.715428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:14.715905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:14.715428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:14.715905 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:14.715567    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:14.715905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:14.715617    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:14.715905 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:14.715742    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:14.715905 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:14.715826    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:15.822036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:15.822001    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gb6tk" event={"ID":"4e6cff18-a46a-424e-9f04-d3339ce97e70","Type":"ContainerStarted","Data":"fd87c8d3eb3095ac7f16d76a289fa570bd54e3c381c73a543ed5b48dc7e1f7b4"}
Jul 06 15:17:15.836019 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:15.835971    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gb6tk" podStartSLOduration=4.394020315 podStartE2EDuration="15.835954176s" totalImagesPullingTime="11.441933861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:15.835819196 +0000 UTC m=+16.616248015" watchObservedRunningTime="2026-07-06 15:17:15.835954176 +0000 UTC m=+16.616382977"
Jul 06 15:17:16.715488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:16.715443    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:16.715488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:16.715470    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:16.715726 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:16.715456    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:16.715726 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:16.715580    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:16.715726 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:16.715652    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:16.715726 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:16.715712    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:17.355238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:17.355155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:17.355729 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.355353    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:17.355729 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.355442    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:17:33.35541922 +0000 UTC m=+34.135848020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:17.355729 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:17.355359    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:17.355729 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.355457    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:17.355729 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.355509    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:17:33.355497257 +0000 UTC m=+34.135926037 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:17.556378 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:17.556346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:17.556565 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.556511    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:17.556565 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.556533    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:17.556565 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.556542    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:17.556681 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:17.556606    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:33.556588647 +0000 UTC m=+34.337017439 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:18.716039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:18.715994    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:18.716039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:18.716025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:18.716633 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:18.716034    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:18.716633 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:18.716109    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:18.716633 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:18.716216    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:18.716633 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:18.716296    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:20.715556 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:20.715520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:20.716007 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:20.715520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:20.716007 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:20.715653    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:20.716007 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:20.715525    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:20.716007 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:20.715750    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:20.716007 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:20.715855    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:21.836604 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.836135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fjdfn" event={"ID":"d8533689-82cc-471a-a23d-17f5d0ab5af5","Type":"ContainerStarted","Data":"d4b7c572ee95fb298ad94ee75001db7d2698a56cbd70e7c3acd5080f241b9ea8"}
Jul 06 15:17:21.850657 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.850622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-74wlp" event={"ID":"fb6f9482-c85f-4d8c-952a-d3f575143f09","Type":"ContainerStarted","Data":"eda98abb14d58549302132b4e6c357708352ef4b525496b107460b67cdab5ac7"}
Jul 06 15:17:21.856700 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.856603    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tpfz6" event={"ID":"9a328533-20cf-48c0-bab3-da113fcaa421","Type":"ContainerStarted","Data":"0998a0b9f3fea8f148aa826a44649e687d7edaf396ae18d81c79bc79724177c0"}
Jul 06 15:17:21.869959 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.869904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fjdfn" podStartSLOduration=10.390415781 podStartE2EDuration="21.869885236s" totalImagesPullingTime="11.479469455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:21.854169576 +0000 UTC m=+22.634598378" watchObservedRunningTime="2026-07-06 15:17:21.869885236 +0000 UTC m=+22.650314037"
Jul 06 15:17:21.890135 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.890043    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-74wlp" podStartSLOduration=10.422466782 podStartE2EDuration="21.890020158s" totalImagesPullingTime="11.467553376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:21.86958907 +0000 UTC m=+22.650017894" watchObservedRunningTime="2026-07-06 15:17:21.890020158 +0000 UTC m=+22.670448961"
Jul 06 15:17:21.928194 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.927874    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hhx2" podStartSLOduration=10.433222094 podStartE2EDuration="21.927851817s" totalImagesPullingTime="11.494629723s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:21.890238505 +0000 UTC m=+22.670667300" watchObservedRunningTime="2026-07-06 15:17:21.927851817 +0000 UTC m=+22.708280679"
Jul 06 15:17:21.928386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:21.928321    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-tpfz6" podStartSLOduration=2.177304803 podStartE2EDuration="21.928306867s" totalImagesPullingTime="19.751002064s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:21.927566246 +0000 UTC m=+22.707995039" watchObservedRunningTime="2026-07-06 15:17:21.928306867 +0000 UTC m=+22.708735672"
Jul 06 15:17:22.715271 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.715242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:22.715472 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.715242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:22.715472 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:22.715341    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:22.715472 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:22.715426    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:22.715472 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.715242    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:22.715619 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:22.715507    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:22.862529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.862486    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="46704a9405d39df60e532a635fb50aa5b6632a94ecdf11e11090d173aaf87a2a" exitCode=0
Jul 06 15:17:22.863090 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.862583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"46704a9405d39df60e532a635fb50aa5b6632a94ecdf11e11090d173aaf87a2a"}
Jul 06 15:17:22.864360 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.864329    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b" event={"ID":"7b72f750-9a65-424c-8930-54664350805b","Type":"ContainerStarted","Data":"be547f15f7caed463442c3e6a2f87c429363c0fe28750bdf3e6c6418aa595bf9"}
Jul 06 15:17:22.866002 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.865974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-d7dql" event={"ID":"fc86a757-c267-46ce-8f42-b3428abd7517","Type":"ContainerStarted","Data":"6b4dac622db9ca4aa4b317261598162c4a9b5269b4526c449665e735f4334fc0"}
Jul 06 15:17:22.867747 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.867621    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hhx2" event={"ID":"c6eb4b5c-48c6-461d-8350-a42e867d4715","Type":"ContainerStarted","Data":"1b54e2ae44401530e2919d64ebd4b4cee3c81d2115a90afd6babc318b7b84b7d"}
Jul 06 15:17:22.870917 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.870841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"0f00895ed315495a3ba601b67e04ac33e3538ed67fc6fa5fee8ffbaa89ebf77d"}
Jul 06 15:17:22.871031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.870921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"d90fbbba7950ce9e84a6165a59ba25db3c0b777b260e94c757f940d0ffc027ee"}
Jul 06 15:17:22.871031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.870960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"798495b9561dc1e6acd8b93e37ab33d395dd2edd75889e8716b61eba59775971"}
Jul 06 15:17:22.871031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.871009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"1558d2064ceb7960f098bb6296d7b3a2d28a92e719b110514b7de5632deb259e"}
Jul 06 15:17:22.871196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.871064    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"f523396005d5d8f5ea45c3639dcb28b4fcd054844259457c38b996aa9c6540c3"}
Jul 06 15:17:22.871196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.871113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"f164ba1e76f1895e29c886f14bf76598545bbc8fc5a15ab5ba10fee195849578"}
Jul 06 15:17:22.919501 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:22.919411    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-d7dql" podStartSLOduration=11.438976719 podStartE2EDuration="22.919381612s" totalImagesPullingTime="11.480404893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:22.918949152 +0000 UTC m=+23.699377957" watchObservedRunningTime="2026-07-06 15:17:22.919381612 +0000 UTC m=+23.699810409"
Jul 06 15:17:23.335570 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:23.335545    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 06 15:17:23.724457 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:23.724339    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T15:17:23.335565537Z","UUID":"d2d493d2-319c-4690-87bf-0b4065b8129c","Handler":null,"Name":"","Endpoint":""}
Jul 06 15:17:23.725903 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:23.725881    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 06 15:17:23.726042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:23.725926    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 06 15:17:23.875182 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:23.875139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b" event={"ID":"7b72f750-9a65-424c-8930-54664350805b","Type":"ContainerStarted","Data":"4b74c25eff800965069b75a8b64c54f55a7432fd7ef789c51938efcbac113d34"}
Jul 06 15:17:24.240721 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.240686    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:24.241434 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.241411    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:24.715931 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.715895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:24.716101 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.715990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:24.716101 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:24.715996    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:24.716101 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.716021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:24.716198 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:24.716110    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:24.716198 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:24.716174    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:24.880794 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.880758    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"ff64f01dd353a500541b88c25606488471bb758b5dec4f6279314a9718e6fb9b"}
Jul 06 15:17:24.881221 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.880884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:24.881493 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:24.881476    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-74wlp"
Jul 06 15:17:25.886144 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:25.886106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b" event={"ID":"7b72f750-9a65-424c-8930-54664350805b","Type":"ContainerStarted","Data":"acc74f010fc24eb1d2b9461c61c6cf0ffe23d8a035ef6a5e9723af2398a4a97b"}
Jul 06 15:17:25.905839 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:25.905788    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-c925b" podStartSLOduration=11.650158995 podStartE2EDuration="25.905770831s" totalImagesPullingTime="14.255611836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:25.90547585 +0000 UTC m=+26.685904653" watchObservedRunningTime="2026-07-06 15:17:25.905770831 +0000 UTC m=+26.686199632"
Jul 06 15:17:26.716083 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:26.716056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:26.716252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:26.716056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:26.716252 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:26.716208    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:26.716364 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:26.716266    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:26.716364 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:26.716065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:26.716364 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:26.716358    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:26.892810 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:26.892779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" event={"ID":"0614eb8a-09a0-4524-9168-99ea0a851c7b","Type":"ContainerStarted","Data":"78c9db6cae594b09e20973c4a45112618b312e936043a912c8934cbcd87f8171"}
Jul 06 15:17:26.924251 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:26.924205    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g" podStartSLOduration=7.128981326 podStartE2EDuration="26.924188721s" totalImagesPullingTime="19.795207395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:26.923449579 +0000 UTC m=+27.703878400" watchObservedRunningTime="2026-07-06 15:17:26.924188721 +0000 UTC m=+27.704617522"
Jul 06 15:17:27.895717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:27.895683    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:27.896189 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:27.895801    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:27.896189 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:27.895848    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:27.911979 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:27.911953    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:27.912137 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:27.912121    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:17:28.715849 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.715651    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:28.716056 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.715649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:28.716056 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.715956    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:28.716056 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.715647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:28.716056 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.716006    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:28.716260 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.716119    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:28.749293 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.749262    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l2l"]
Jul 06 15:17:28.751763 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.751739    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rrbh"]
Jul 06 15:17:28.754653 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.754630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tg88d"]
Jul 06 15:17:28.897023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.896991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:28.897023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.897008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:28.897580 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:28.897013    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:28.897580 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.897106    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:28.897580 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.897437    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:28.897580 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:28.897498    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:29.900530 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:29.900498    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="285424dc1bb33f278b229a66afdd86f231bb61979f9baab08056c55642dc04f0" exitCode=0
Jul 06 15:17:29.900993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:29.900579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"285424dc1bb33f278b229a66afdd86f231bb61979f9baab08056c55642dc04f0"}
Jul 06 15:17:30.715989 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:30.715955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:30.715989 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:30.715955    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:30.716185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:30.715973    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:30.716185 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:30.716082    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:30.716254 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:30.716223    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:30.716356 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:30.716336    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:31.908715 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:31.908685    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="865f9c7df7f8e25b1790f5087a6f2b019dbfad525f321711c85532d193681e5a" exitCode=0
Jul 06 15:17:31.909132 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:31.908722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"865f9c7df7f8e25b1790f5087a6f2b019dbfad525f321711c85532d193681e5a"}
Jul 06 15:17:32.715671 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:32.715639    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:32.715835 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:32.715637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:32.715835 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:32.715769    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:17:32.715835 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:32.715637    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:32.715835 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:32.715824    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tg88d" podUID="bb7546ef-7011-4913-a1da-7d60945f6914"
Jul 06 15:17:32.715965 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:32.715886    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-92l2l" podUID="58a9d157-df15-4431-bd46-f4a4f3e3ea2d"
Jul 06 15:17:32.913216 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:32.913184    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="d3354a4df12bafd902edaf524fcc5f0afb0b6acea5b571e3777011f90d3d39b4" exitCode=0
Jul 06 15:17:32.913574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:32.913222    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"d3354a4df12bafd902edaf524fcc5f0afb0b6acea5b571e3777011f90d3d39b4"}
Jul 06 15:17:33.377202 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:33.377011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:33.377458 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:33.377220    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:33.377458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.377158    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:33.377458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.377304    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:33.377458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.377325    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:18:05.377305952 +0000 UTC m=+66.157734751 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 06 15:17:33.377458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.377342    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret podName:58a9d157-df15-4431-bd46-f4a4f3e3ea2d nodeName:}" failed. No retries permitted until 2026-07-06 15:18:05.377331148 +0000 UTC m=+66.157759964 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret") pod "global-pull-secret-syncer-92l2l" (UID: "58a9d157-df15-4431-bd46-f4a4f3e3ea2d") : object "kube-system"/"original-pull-secret" not registered
Jul 06 15:17:33.579625 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:33.579587    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:33.579823 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.579734    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 06 15:17:33.579823 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.579754    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 06 15:17:33.579823 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.579766    2566 projected.go:196] Error preparing data for projected volume kube-api-access-2xg7k for pod openshift-network-diagnostics/network-check-target-tg88d: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:33.579823 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:33.579822    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k podName:bb7546ef-7011-4913-a1da-7d60945f6914 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:05.579805756 +0000 UTC m=+66.360234539 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2xg7k" (UniqueName: "kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k") pod "network-check-target-tg88d" (UID: "bb7546ef-7011-4913-a1da-7d60945f6914") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 06 15:17:34.483483 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.483383    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-27.ec2.internal" event="NodeReady"
Jul 06 15:17:34.483842 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.483535    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 06 15:17:34.547012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.546976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:17:34.565809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.565756    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.574431 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.574120    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 06 15:17:34.574431 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.574153    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7t2f5\""
Jul 06 15:17:34.574431 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.574219    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 06 15:17:34.574946 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.574928    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 06 15:17:34.577676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.577656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:17:34.581691 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.581513    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 06 15:17:34.582050 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.582032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"]
Jul 06 15:17:34.601151 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.601110    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jjtrj"]
Jul 06 15:17:34.601314 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.601244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.604958 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.604927    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 06 15:17:34.605165 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.605097    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 06 15:17:34.605736 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.605045    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 06 15:17:34.606095 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.606073    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 06 15:17:34.606510 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.606464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 06 15:17:34.606747 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.606657    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 06 15:17:34.608764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.608741    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 06 15:17:34.616875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.616840    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"]
Jul 06 15:17:34.618292 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.618268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.622496 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.622471    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 06 15:17:34.622630 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.622483    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-x9pws\""
Jul 06 15:17:34.622719 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.622706    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 06 15:17:34.623096 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.622966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 06 15:17:34.631145 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.631113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-vlx8d"]
Jul 06 15:17:34.631299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.631255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.634308 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.634286    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 06 15:17:34.649471 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.649427    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"]
Jul 06 15:17:34.649639 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.649579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.652257 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.652239    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\""
Jul 06 15:17:34.653905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.653890    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 06 15:17:34.653905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.653902    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 06 15:17:34.675369 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675334    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjtrj"]
Jul 06 15:17:34.675369 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"]
Jul 06 15:17:34.675576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675409    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"]
Jul 06 15:17:34.675576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675423    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"]
Jul 06 15:17:34.675576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlx8d"]
Jul 06 15:17:34.675576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.675469    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.678410 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.678367    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 06 15:17:34.678524 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.678494    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-97bxr\""
Jul 06 15:17:34.687556 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.687662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687583    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5sdq\" (UniqueName: \"kubernetes.io/projected/6d537d6b-7607-4d25-b316-d98f263290cb-kube-api-access-h5sdq\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.687662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d537d6b-7607-4d25-b316-d98f263290cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.687764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687667    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687739    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.687826 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687764    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfg4q\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687826 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687784    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687826 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687914 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687914 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.687975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687956    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.688036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.687975    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.715158 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.715125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:17:34.715321 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.715228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:17:34.715321 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.715251    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:17:34.718140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718114    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 15:17:34.718288 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718239    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 15:17:34.718358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\""
Jul 06 15:17:34.718546 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 15:17:34.718783 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718581    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\""
Jul 06 15:17:34.718783 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.718572    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 15:17:34.788686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.788686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84649a02-593c-4194-905b-14bada246537-tmp-dir\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/955cc9f3-7e5f-4c19-a572-beb2761d9f66-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788769    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5sdq\" (UniqueName: \"kubernetes.io/projected/6d537d6b-7607-4d25-b316-d98f263290cb-kube-api-access-h5sdq\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788861    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-cert\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.788937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788902    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d537d6b-7607-4d25-b316-d98f263290cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788931    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58ab71e-82eb-4a12-9b81-27d326fee966-tmp\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.788993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84649a02-593c-4194-905b-14bada246537-config-volume\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789042    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7sv\" (UniqueName: \"kubernetes.io/projected/84649a02-593c-4194-905b-14bada246537-kube-api-access-lf7sv\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789132    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b58ab71e-82eb-4a12-9b81-27d326fee966-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/955cc9f3-7e5f-4c19-a572-beb2761d9f66-kube-api-access-mjx6z\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfg4q\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789228    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fw7\" (UniqueName: \"kubernetes.io/projected/b58ab71e-82eb-4a12-9b81-27d326fee966-kube-api-access-x5fw7\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrnkv\" (UniqueName: \"kubernetes.io/projected/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-kube-api-access-jrnkv\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789432    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789513    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:34.789586    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:34.789608    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:34.789734 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:34.789667    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:17:35.289646842 +0000 UTC m=+36.070075649 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:34.790173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.789822    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.790173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.790016    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.790470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.790443    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.790605 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.790580    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d537d6b-7607-4d25-b316-d98f263290cb-additional-service-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.794327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.794282    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.794478 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.794368    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.794478 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.794453    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.794595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.794519    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.794906 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.794880    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.795503 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.795482    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6d537d6b-7607-4d25-b316-d98f263290cb-ca\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.799773 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.799728    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5sdq\" (UniqueName: \"kubernetes.io/projected/6d537d6b-7607-4d25-b316-d98f263290cb-kube-api-access-h5sdq\") pod \"cluster-proxy-proxy-agent-699b8646fb-xsfxg\" (UID: \"6d537d6b-7607-4d25-b316-d98f263290cb\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.799773 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.799766    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfg4q\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.812273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.812237    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:34.890641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58ab71e-82eb-4a12-9b81-27d326fee966-tmp\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.890641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84649a02-593c-4194-905b-14bada246537-config-volume\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.890865 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7sv\" (UniqueName: \"kubernetes.io/projected/84649a02-593c-4194-905b-14bada246537-kube-api-access-lf7sv\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.890865 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b58ab71e-82eb-4a12-9b81-27d326fee966-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.890975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/955cc9f3-7e5f-4c19-a572-beb2761d9f66-kube-api-access-mjx6z\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.890975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fw7\" (UniqueName: \"kubernetes.io/projected/b58ab71e-82eb-4a12-9b81-27d326fee966-kube-api-access-x5fw7\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.890975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890914    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrnkv\" (UniqueName: \"kubernetes.io/projected/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-kube-api-access-jrnkv\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.890975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890965    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.891170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.890981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84649a02-593c-4194-905b-14bada246537-tmp-dir\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.891170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.891028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/955cc9f3-7e5f-4c19-a572-beb2761d9f66-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.891170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.891061    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b58ab71e-82eb-4a12-9b81-27d326fee966-tmp\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.891170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.891079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-cert\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.891336 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:34.891210    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:34.891336 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:34.891275    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:35.391256707 +0000 UTC m=+36.171685505 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:34.891660 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.891629    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/84649a02-593c-4194-905b-14bada246537-config-volume\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.891862 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.891840    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84649a02-593c-4194-905b-14bada246537-tmp-dir\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.894092 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.894071    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-cert\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.894092 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.894079    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b58ab71e-82eb-4a12-9b81-27d326fee966-klusterlet-config\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.894221 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.894092    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/955cc9f3-7e5f-4c19-a572-beb2761d9f66-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.902099 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.901255    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fw7\" (UniqueName: \"kubernetes.io/projected/b58ab71e-82eb-4a12-9b81-27d326fee966-kube-api-access-x5fw7\") pod \"klusterlet-addon-workmgr-7dbcffdbf7-x2d9l\" (UID: \"b58ab71e-82eb-4a12-9b81-27d326fee966\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.902099 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.901426    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx6z\" (UniqueName: \"kubernetes.io/projected/955cc9f3-7e5f-4c19-a572-beb2761d9f66-kube-api-access-mjx6z\") pod \"managed-serviceaccount-addon-agent-6898f4959b-74rh9\" (UID: \"955cc9f3-7e5f-4c19-a572-beb2761d9f66\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:34.903223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.903202    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7sv\" (UniqueName: \"kubernetes.io/projected/84649a02-593c-4194-905b-14bada246537-kube-api-access-lf7sv\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:34.904154 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.904132    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrnkv\" (UniqueName: \"kubernetes.io/projected/0dcfb475-f876-4ebc-927e-ea7e5d4a450c-kube-api-access-jrnkv\") pod \"ingress-canary-jjtrj\" (UID: \"0dcfb475-f876-4ebc-927e-ea7e5d4a450c\") " pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.924821 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.924782    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:17:34.932923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.932895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jjtrj"
Jul 06 15:17:34.943143 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.943116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:34.984148 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:34.984122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"
Jul 06 15:17:35.204562 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.204526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jjtrj"]
Jul 06 15:17:35.210523 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:35.210497    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcfb475_f876_4ebc_927e_ea7e5d4a450c.slice/crio-251d18c35fd0f93593a0ff66c413fe75a4a52e51782e80205ff3dde048500694 WatchSource:0}: Error finding container 251d18c35fd0f93593a0ff66c413fe75a4a52e51782e80205ff3dde048500694: Status 404 returned error can't find the container with id 251d18c35fd0f93593a0ff66c413fe75a4a52e51782e80205ff3dde048500694
Jul 06 15:17:35.231856 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:35.231637    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb58ab71e_82eb_4a12_9b81_27d326fee966.slice/crio-99ff92e82691f10aa39c05d3fd4fcca1c5c03a5dd266bb70f5a85ae08597a341 WatchSource:0}: Error finding container 99ff92e82691f10aa39c05d3fd4fcca1c5c03a5dd266bb70f5a85ae08597a341: Status 404 returned error can't find the container with id 99ff92e82691f10aa39c05d3fd4fcca1c5c03a5dd266bb70f5a85ae08597a341
Jul 06 15:17:35.232466 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:35.232440    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d537d6b_7607_4d25_b316_d98f263290cb.slice/crio-396d4c95122086fced0e839017466f41fb63ea075b9fd0efd68faddc3917c61b WatchSource:0}: Error finding container 396d4c95122086fced0e839017466f41fb63ea075b9fd0efd68faddc3917c61b: Status 404 returned error can't find the container with id 396d4c95122086fced0e839017466f41fb63ea075b9fd0efd68faddc3917c61b
Jul 06 15:17:35.232818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.232800    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"]
Jul 06 15:17:35.233212 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:17:35.233182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod955cc9f3_7e5f_4c19_a572_beb2761d9f66.slice/crio-791d55baa3d9bc17907ed143592b49216bd5675a0afac4d888691a4226f2eb11 WatchSource:0}: Error finding container 791d55baa3d9bc17907ed143592b49216bd5675a0afac4d888691a4226f2eb11: Status 404 returned error can't find the container with id 791d55baa3d9bc17907ed143592b49216bd5675a0afac4d888691a4226f2eb11
Jul 06 15:17:35.233966 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.233946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"]
Jul 06 15:17:35.234691 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.234672    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9"]
Jul 06 15:17:35.286602 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.286381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:35.295409 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.295362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:35.295583 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:35.295506    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:35.295583 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:35.295521    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:35.295757 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:35.295588    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:17:36.295567495 +0000 UTC m=+37.075996286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:35.396586 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.396545    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:35.396770 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:35.396719    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:35.396822 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:35.396792    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:36.396773134 +0000 UTC m=+37.177201917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:35.422889 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.422852    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:35.523270 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.523242    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:35.592004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.591965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:35.674814 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.674779    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:35.923625 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.923540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9" event={"ID":"955cc9f3-7e5f-4c19-a572-beb2761d9f66","Type":"ContainerStarted","Data":"791d55baa3d9bc17907ed143592b49216bd5675a0afac4d888691a4226f2eb11"}
Jul 06 15:17:35.925329 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.925299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" event={"ID":"b58ab71e-82eb-4a12-9b81-27d326fee966","Type":"ContainerStarted","Data":"99ff92e82691f10aa39c05d3fd4fcca1c5c03a5dd266bb70f5a85ae08597a341"}
Jul 06 15:17:35.927169 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.927141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjtrj" event={"ID":"0dcfb475-f876-4ebc-927e-ea7e5d4a450c","Type":"ContainerStarted","Data":"251d18c35fd0f93593a0ff66c413fe75a4a52e51782e80205ff3dde048500694"}
Jul 06 15:17:35.928948 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:35.928922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerStarted","Data":"396d4c95122086fced0e839017466f41fb63ea075b9fd0efd68faddc3917c61b"}
Jul 06 15:17:36.305066 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.304328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:36.305066 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:36.304588    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:36.305066 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:36.304605    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:36.305066 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:36.304664    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:17:38.304646702 +0000 UTC m=+39.085075487 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:36.323497 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.323445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:36.405086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.405003    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:36.405279 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:36.405208    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:36.405279 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:36.405273    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:38.405252539 +0000 UTC m=+39.185681320 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:36.417245 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.417212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:36.639289 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.639255    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:36.743081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:36.743049    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:37.679417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:37.679353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:37.793247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:37.793206    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:37.930276 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:37.930138    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:38.323495 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:38.322495    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:38.323495 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:38.322683    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:38.323495 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:38.322707    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:38.323495 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:38.322784    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:17:42.322763117 +0000 UTC m=+43.103191897 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:38.423930 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:38.423791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:38.424121 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:38.423944    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:38.424121 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:38.424006    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:42.423989513 +0000 UTC m=+43.204418296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:39.958374 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:39.958331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:42.361523 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:42.361483    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:42.361930 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:42.361656    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:42.361930 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:42.361674    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:42.361930 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:42.361730    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:17:50.361713447 +0000 UTC m=+51.142142226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:42.461967 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:42.461925    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:42.462160 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:42.462063    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:42.462160 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:42.462134    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:17:50.462115592 +0000 UTC m=+51.242544384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:45.961076 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.961036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerStarted","Data":"33a290666b98531e971d9ab5dadee953030de90ed55b438f2cdd5c451d10e278"}
Jul 06 15:17:45.961589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.961089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerStarted","Data":"e7df97ead351787cebf74567aa79737854016322bcd82c5628dd3a0d690fa850"}
Jul 06 15:17:45.961589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.961106    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerStarted","Data":"0e0c000c62cfe60e635b08d3301bb0e9368693e116efaec38c4acdc8ffcdd965"}
Jul 06 15:17:45.962553 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.962528    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9" event={"ID":"955cc9f3-7e5f-4c19-a572-beb2761d9f66","Type":"ContainerStarted","Data":"942ee3ce3a79225f3a87aa61a1b62c5e54f6416181848bdab2a58757e97f1466"}
Jul 06 15:17:45.963874 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.963854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" event={"ID":"b58ab71e-82eb-4a12-9b81-27d326fee966","Type":"ContainerStarted","Data":"4492c0c89827cea8482e1198c5ee810d1b1c598c3e0cc8d814f11e6240b1236d"}
Jul 06 15:17:45.964024 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.964000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:45.965490 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.965467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jjtrj" event={"ID":"0dcfb475-f876-4ebc-927e-ea7e5d4a450c","Type":"ContainerStarted","Data":"b619d4ca7a7bdf02931cb82e7a4bc18097bc5e764253faa91723a0c021ab9f08"}
Jul 06 15:17:45.965918 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.965897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:17:45.968130 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.968108    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="ec0416361ea61a037863a03cd215303162577a49cf5715909741d4aaa00aca53" exitCode=0
Jul 06 15:17:45.968225 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.968141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"ec0416361ea61a037863a03cd215303162577a49cf5715909741d4aaa00aca53"}
Jul 06 15:17:45.970077 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.970052    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log" limit=1024
Jul 06 15:17:45.987280 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:45.987214    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" podStartSLOduration=33.10493631 podStartE2EDuration="42.987200783s" totalImagesPullingTime="9.882264473s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:45.985841361 +0000 UTC m=+46.766270154" watchObservedRunningTime="2026-07-06 15:17:45.987200783 +0000 UTC m=+46.767629588"
Jul 06 15:17:46.029293 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:46.029207    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jjtrj" podStartSLOduration=2.123861305 podStartE2EDuration="12.029183163s" totalImagesPullingTime="9.905321858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:46.027891728 +0000 UTC m=+46.808320542" watchObservedRunningTime="2026-07-06 15:17:46.029183163 +0000 UTC m=+46.809611967"
Jul 06 15:17:46.054829 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:46.054777    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" podStartSLOduration=33.157628018 podStartE2EDuration="43.054760701s" totalImagesPullingTime="9.897132683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:46.054142297 +0000 UTC m=+46.834571101" watchObservedRunningTime="2026-07-06 15:17:46.054760701 +0000 UTC m=+46.835189501"
Jul 06 15:17:46.082714 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:46.082665    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9" podStartSLOduration=33.201032652 podStartE2EDuration="43.08264643s" totalImagesPullingTime="9.881613778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:46.081950277 +0000 UTC m=+46.862379078" watchObservedRunningTime="2026-07-06 15:17:46.08264643 +0000 UTC m=+46.863075209"
Jul 06 15:17:46.973796 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:46.973759    2566 generic.go:356] "Generic (PLEG): container finished" podID="c6ab13cd-e191-471b-ada3-cf9055c0abc2" containerID="92b5d97ff335d61e2fcef834e7244ec01cb67c6816a4be3bdb5aac3d2a9687c9" exitCode=0
Jul 06 15:17:46.974191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:46.973859    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerDied","Data":"92b5d97ff335d61e2fcef834e7244ec01cb67c6816a4be3bdb5aac3d2a9687c9"}
Jul 06 15:17:47.981135 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:47.981098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" event={"ID":"c6ab13cd-e191-471b-ada3-cf9055c0abc2","Type":"ContainerStarted","Data":"a4f5b68bf02cb61cfd9a8605a044166ce6d6f733454606782c53bb5597dc2e4a"}
Jul 06 15:17:48.005947 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:48.005895    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mbwwf" podStartSLOduration=15.836879208 podStartE2EDuration="48.005879479s" totalImagesPullingTime="32.169000271s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:17:48.004776077 +0000 UTC m=+48.785204889" watchObservedRunningTime="2026-07-06 15:17:48.005879479 +0000 UTC m=+48.786308276"
Jul 06 15:17:50.422846 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:50.422805    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:17:50.423260 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:50.422959    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:17:50.423260 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:50.422981    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:17:50.423260 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:50.423040    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:18:06.423024128 +0000 UTC m=+67.203452911 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:17:50.523639 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:50.523593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:17:50.523821 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:50.523742    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:17:50.523821 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:17:50.523812    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:06.523796771 +0000 UTC m=+67.304225581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:17:59.911015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:17:59.910988    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-tsn7g"
Jul 06 15:18:05.430012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.429970    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:18:05.430537 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.430072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:18:05.432762 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.432743    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 06 15:18:05.433070 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.433055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 06 15:18:05.441054 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:05.441027    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 15:18:05.441204 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:05.441099    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:19:09.441079513 +0000 UTC m=+130.221508305 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : secret "metrics-daemon-secret" not found
Jul 06 15:18:05.444099 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.444063    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/58a9d157-df15-4431-bd46-f4a4f3e3ea2d-original-pull-secret\") pod \"global-pull-secret-syncer-92l2l\" (UID: \"58a9d157-df15-4431-bd46-f4a4f3e3ea2d\") " pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:18:05.632665 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.632615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:18:05.635790 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.635769    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 06 15:18:05.645609 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.645585    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 06 15:18:05.649905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.649889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-92l2l"
Jul 06 15:18:05.655744 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.655725    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xg7k\" (UniqueName: \"kubernetes.io/projected/bb7546ef-7011-4913-a1da-7d60945f6914-kube-api-access-2xg7k\") pod \"network-check-target-tg88d\" (UID: \"bb7546ef-7011-4913-a1da-7d60945f6914\") " pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:18:05.777426 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.777379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-92l2l"]
Jul 06 15:18:05.780570 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:18:05.780533    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a9d157_df15_4431_bd46_f4a4f3e3ea2d.slice/crio-faffdb3b41d03df88150217c23211bf0228481dfc4fb56159aae77761c73e4b7 WatchSource:0}: Error finding container faffdb3b41d03df88150217c23211bf0228481dfc4fb56159aae77761c73e4b7: Status 404 returned error can't find the container with id faffdb3b41d03df88150217c23211bf0228481dfc4fb56159aae77761c73e4b7
Jul 06 15:18:05.946651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.946620    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-x2rt6\""
Jul 06 15:18:05.954149 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:05.954082    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:18:06.040422 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:06.040375    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l2l" event={"ID":"58a9d157-df15-4431-bd46-f4a4f3e3ea2d","Type":"ContainerStarted","Data":"faffdb3b41d03df88150217c23211bf0228481dfc4fb56159aae77761c73e4b7"}
Jul 06 15:18:06.104410 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:06.104358    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tg88d"]
Jul 06 15:18:06.118340 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:18:06.118311    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb7546ef_7011_4913_a1da_7d60945f6914.slice/crio-bd1b465fdf60c7f6de28979962797e4bec25dbee4a88f413defe976d243c7df6 WatchSource:0}: Error finding container bd1b465fdf60c7f6de28979962797e4bec25dbee4a88f413defe976d243c7df6: Status 404 returned error can't find the container with id bd1b465fdf60c7f6de28979962797e4bec25dbee4a88f413defe976d243c7df6
Jul 06 15:18:06.439107 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:06.439071    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:18:06.439590 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:06.439194    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:18:06.439590 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:06.439210    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:18:06.439590 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:06.439269    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:18:38.439255356 +0000 UTC m=+99.219684135 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:18:06.539852 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:06.539812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:18:06.540031 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:06.539998    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:18:06.540102 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:06.540077    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:18:38.540056088 +0000 UTC m=+99.320484886 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:18:07.045297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:07.045250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tg88d" event={"ID":"bb7546ef-7011-4913-a1da-7d60945f6914","Type":"ContainerStarted","Data":"bd1b465fdf60c7f6de28979962797e4bec25dbee4a88f413defe976d243c7df6"}
Jul 06 15:18:12.065496 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:12.065407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-92l2l" event={"ID":"58a9d157-df15-4431-bd46-f4a4f3e3ea2d","Type":"ContainerStarted","Data":"bac6b267961cfd1731306c7fd73814c83016394b8befd1ff4b5921ec923a7d59"}
Jul 06 15:18:12.066793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:12.066772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tg88d" event={"ID":"bb7546ef-7011-4913-a1da-7d60945f6914","Type":"ContainerStarted","Data":"3b9cf9c996b9ab9def4dc3442da7ad3aee3e48bbff1b00ede3b34471e6095a81"}
Jul 06 15:18:12.066916 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:12.066901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:18:12.084878 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:12.084832    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-92l2l" podStartSLOduration=66.148189337 podStartE2EDuration="1m12.084815747s" totalImagesPullingTime="5.93662641s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:12.084122234 +0000 UTC m=+72.864551045" watchObservedRunningTime="2026-07-06 15:18:12.084815747 +0000 UTC m=+72.865244550"
Jul 06 15:18:12.105477 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:12.105437    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tg88d" podStartSLOduration=66.496990336 podStartE2EDuration="1m12.105423626s" totalImagesPullingTime="5.60843329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:18:12.10429108 +0000 UTC m=+72.884719881" watchObservedRunningTime="2026-07-06 15:18:12.105423626 +0000 UTC m=+72.885852426"
Jul 06 15:18:38.467271 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:38.467230    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:18:38.467714 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:38.467434    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 06 15:18:38.467714 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:38.467457    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-767857c9d5-7vfj6: secret "image-registry-tls" not found
Jul 06 15:18:38.467714 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:38.467531    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls podName:2edfb766-da3c-4db0-8791-81bbe22dfc3f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:42.467515187 +0000 UTC m=+163.247943986 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls") pod "image-registry-767857c9d5-7vfj6" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f") : secret "image-registry-tls" not found
Jul 06 15:18:38.568075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:38.567997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:18:38.568220 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:38.568150    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 06 15:18:38.568260 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:18:38.568226    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls podName:84649a02-593c-4194-905b-14bada246537 nodeName:}" failed. No retries permitted until 2026-07-06 15:19:42.568210543 +0000 UTC m=+163.348639336 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls") pod "dns-default-vlx8d" (UID: "84649a02-593c-4194-905b-14bada246537") : secret "dns-default-metrics-tls" not found
Jul 06 15:18:43.072049 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:18:43.072010    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tg88d"
Jul 06 15:19:00.036727 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.036698    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-jv75h"]
Jul 06 15:19:00.038541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.038526    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.064703 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.064674    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-tls\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret"
Jul 06 15:19:00.065961 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.065938    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 06 15:19:00.066432 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.066408    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"insights-runtime-extractor-jv75h\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="f7127008-cf84-40d0-8d43-e36e77c3608f" pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.066555 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.066538    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-rbac-proxy\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap"
Jul 06 15:19:00.067083 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.067063    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"insights-runtime-extractor-sa-dockercfg-rqfh6\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqfh6\"" type="*v1.Secret"
Jul 06 15:19:00.067213 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:00.067193    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-insights\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 06 15:19:00.084271 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.084244    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jv75h"]
Jul 06 15:19:00.132554 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.132521    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.132554 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.132559    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7127008-cf84-40d0-8d43-e36e77c3608f-crio-socket\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.132732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.132578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7127008-cf84-40d0-8d43-e36e77c3608f-data-volume\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.132732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.132594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.132732 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.132718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9nfd\" (UniqueName: \"kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.233798 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.233798 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7127008-cf84-40d0-8d43-e36e77c3608f-crio-socket\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.234028 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7127008-cf84-40d0-8d43-e36e77c3608f-data-volume\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.234028 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.234028 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nfd\" (UniqueName: \"kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.234028 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.233908    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/f7127008-cf84-40d0-8d43-e36e77c3608f-crio-socket\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:00.234168 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:00.234111    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/f7127008-cf84-40d0-8d43-e36e77c3608f-data-volume\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:01.003108 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.003072    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 06 15:19:01.044634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.044604    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 06 15:19:01.054330 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.054304    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:19:01.057710 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.054696    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:01.554669398 +0000 UTC m=+122.335098181 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : configmap "kube-rbac-proxy" not found
Jul 06 15:19:01.234825 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.234796    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: failed to sync secret cache: timed out waiting for the condition
Jul 06 15:19:01.235012 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.234874    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:01.734853516 +0000 UTC m=+122.515282305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : failed to sync secret cache: timed out waiting for the condition
Jul 06 15:19:01.240945 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.240916    2566 projected.go:291] Couldn't get configMap openshift-insights/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Jul 06 15:19:01.241069 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.240952    2566 projected.go:196] Error preparing data for projected volume kube-api-access-r9nfd for pod openshift-insights/insights-runtime-extractor-jv75h: failed to sync configmap cache: timed out waiting for the condition
Jul 06 15:19:01.241069 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.241002    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:01.740988989 +0000 UTC m=+122.521417782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r9nfd" (UniqueName: "kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : failed to sync configmap cache: timed out waiting for the condition
Jul 06 15:19:01.243446 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.243428    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rqfh6\""
Jul 06 15:19:01.300360 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.300282    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 06 15:19:01.604446 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.604425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 06 15:19:01.645513 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.645474    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:01.645678 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.645586    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:19:01.645678 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.645646    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:02.645631568 +0000 UTC m=+123.426060347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : configmap "kube-rbac-proxy" not found
Jul 06 15:19:01.746477 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.746444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:01.746618 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.746510    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9nfd\" (UniqueName: \"kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:01.746618 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.746592    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:01.746685 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:01.746646    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:02.746631329 +0000 UTC m=+123.527060107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:01.748722 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.748695    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9nfd\" (UniqueName: \"kubernetes.io/projected/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-api-access-r9nfd\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:01.807187 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:01.807161    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7dql_fc86a757-c267-46ce-8f42-b3428abd7517/dns-node-resolver/0.log"
Jul 06 15:19:02.653966 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:02.653927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:02.654458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:02.654039    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:19:02.654458 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:02.654111    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:04.654095642 +0000 UTC m=+125.434524455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : configmap "kube-rbac-proxy" not found
Jul 06 15:19:02.754710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:02.754669    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:02.754850 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:02.754827    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:02.754913 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:02.754902    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:04.754884105 +0000 UTC m=+125.535312907 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:02.807879 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:02.807857    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hhx2_c6eb4b5c-48c6-461d-8350-a42e867d4715/node-ca/0.log"
Jul 06 15:19:03.606489 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:03.606455    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjtrj_0dcfb475-f876-4ebc-927e-ea7e5d4a450c/serve-healthcheck-canary/0.log"
Jul 06 15:19:04.668129 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:04.668090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:04.668562 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:04.668217    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:19:04.668562 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:04.668277    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:08.668262674 +0000 UTC m=+129.448691462 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : configmap "kube-rbac-proxy" not found
Jul 06 15:19:04.768716 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:04.768671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:04.768899 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:04.768797    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:04.768899 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:04.768881    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:08.768860025 +0000 UTC m=+129.549288832 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:08.704254 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:08.704204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:08.704681 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:08.704329    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 06 15:19:08.704681 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:08.704412    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:16.704379793 +0000 UTC m=+137.484808576 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : configmap "kube-rbac-proxy" not found
Jul 06 15:19:08.805258 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:08.805225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:08.805453 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:08.805349    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:08.805453 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:08.805422    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:16.805404628 +0000 UTC m=+137.585833422 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:09.510548 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:09.510508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:19:09.510730 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:09.510659    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 06 15:19:09.510730 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:09.510721    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs podName:b57e3d86-626d-4705-b410-5694cfd1accd nodeName:}" failed. No retries permitted until 2026-07-06 15:21:11.510706622 +0000 UTC m=+252.291135420 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs") pod "network-metrics-daemon-7rrbh" (UID: "b57e3d86-626d-4705-b410-5694cfd1accd") : secret "metrics-daemon-secret" not found
Jul 06 15:19:16.763554 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:16.763511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:16.764036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:16.764016    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/f7127008-cf84-40d0-8d43-e36e77c3608f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:16.864098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:16.864060    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:16.864236 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:16.864216    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:16.864284 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:16.864274    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls podName:f7127008-cf84-40d0-8d43-e36e77c3608f nodeName:}" failed. No retries permitted until 2026-07-06 15:19:32.864260429 +0000 UTC m=+153.644689211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-jv75h" (UID: "f7127008-cf84-40d0-8d43-e36e77c3608f") : secret "insights-runtime-extractor-tls" not found
Jul 06 15:19:32.887642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:32.887585    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:32.889877 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:32.889842    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/f7127008-cf84-40d0-8d43-e36e77c3608f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-jv75h\" (UID: \"f7127008-cf84-40d0-8d43-e36e77c3608f\") " pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:33.046778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:33.046724    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-jv75h"
Jul 06 15:19:33.172354 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:33.172284    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-jv75h"]
Jul 06 15:19:33.175495 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:19:33.175468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7127008_cf84_40d0_8d43_e36e77c3608f.slice/crio-0b0b0929be2a09df0b5960e4d52d054d12bff3f414b779c16c48b483c0347510 WatchSource:0}: Error finding container 0b0b0929be2a09df0b5960e4d52d054d12bff3f414b779c16c48b483c0347510: Status 404 returned error can't find the container with id 0b0b0929be2a09df0b5960e4d52d054d12bff3f414b779c16c48b483c0347510
Jul 06 15:19:33.302592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:33.302555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jv75h" event={"ID":"f7127008-cf84-40d0-8d43-e36e77c3608f","Type":"ContainerStarted","Data":"0b0b0929be2a09df0b5960e4d52d054d12bff3f414b779c16c48b483c0347510"}
Jul 06 15:19:37.316470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:37.316431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jv75h" event={"ID":"f7127008-cf84-40d0-8d43-e36e77c3608f","Type":"ContainerStarted","Data":"ae9a956bff1d5b28ea962990d05f258a9ff65c8787a62fdc660eccff5edff7ba"}
Jul 06 15:19:37.577597 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:37.577509    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f"
Jul 06 15:19:37.657716 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:37.657686    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-vlx8d" podUID="84649a02-593c-4194-905b-14bada246537"
Jul 06 15:19:37.756406 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:19:37.756355    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7rrbh" podUID="b57e3d86-626d-4705-b410-5694cfd1accd"
Jul 06 15:19:38.321324 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:38.321293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:19:38.321726 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:38.321284    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jv75h" event={"ID":"f7127008-cf84-40d0-8d43-e36e77c3608f","Type":"ContainerStarted","Data":"6439b00394670d007fe20e2c044d5c8d285dd45aadb062529a17e9733d112e53"}
Jul 06 15:19:38.321726 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:38.321475    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:41.333539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:41.333506    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-jv75h" event={"ID":"f7127008-cf84-40d0-8d43-e36e77c3608f","Type":"ContainerStarted","Data":"f02d8646934ef376dd1d75c8eb44f8c14ae8760d234512949dd422cdb910b988"}
Jul 06 15:19:41.351946 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:41.351902    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-jv75h" podStartSLOduration=34.139804359 podStartE2EDuration="41.351886326s" totalImagesPullingTime="7.212081967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 15:19:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:41.350687302 +0000 UTC m=+162.131116131" watchObservedRunningTime="2026-07-06 15:19:41.351886326 +0000 UTC m=+162.132315105"
Jul 06 15:19:42.470212 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.470166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:19:42.472486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.472465    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"image-registry-767857c9d5-7vfj6\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") " pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:19:42.524504 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.524475    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7t2f5\""
Jul 06 15:19:42.532761 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.532743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:19:42.570932 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.570894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:42.573196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.573144    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/84649a02-593c-4194-905b-14bada246537-metrics-tls\") pod \"dns-default-vlx8d\" (UID: \"84649a02-593c-4194-905b-14bada246537\") " pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:42.657371 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.657331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:19:42.661276 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:19:42.661243    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2edfb766_da3c_4db0_8791_81bbe22dfc3f.slice/crio-128fea2608743b54c0e64039a83e6505f62665006bdb7ebe90d1a8bf2f468f09 WatchSource:0}: Error finding container 128fea2608743b54c0e64039a83e6505f62665006bdb7ebe90d1a8bf2f468f09: Status 404 returned error can't find the container with id 128fea2608743b54c0e64039a83e6505f62665006bdb7ebe90d1a8bf2f468f09
Jul 06 15:19:42.826081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.825997    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8l7xg\""
Jul 06 15:19:42.833140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.833121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:42.966181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:42.966149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-vlx8d"]
Jul 06 15:19:42.970462 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:19:42.970378    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84649a02_593c_4194_905b_14bada246537.slice/crio-7b9dc4c9dfb3a67aa3df4c3d70e13fccc27d16b3b5ddf3d10cf86e0f4239fab2 WatchSource:0}: Error finding container 7b9dc4c9dfb3a67aa3df4c3d70e13fccc27d16b3b5ddf3d10cf86e0f4239fab2: Status 404 returned error can't find the container with id 7b9dc4c9dfb3a67aa3df4c3d70e13fccc27d16b3b5ddf3d10cf86e0f4239fab2
Jul 06 15:19:43.341281 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:43.341244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" event={"ID":"2edfb766-da3c-4db0-8791-81bbe22dfc3f","Type":"ContainerStarted","Data":"04e0522edd54e4cffbc99d925400b3366c987fdebf1f685f9a6de93170cddd9a"}
Jul 06 15:19:43.341529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:43.341300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" event={"ID":"2edfb766-da3c-4db0-8791-81bbe22dfc3f","Type":"ContainerStarted","Data":"128fea2608743b54c0e64039a83e6505f62665006bdb7ebe90d1a8bf2f468f09"}
Jul 06 15:19:43.341529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:43.341348    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:19:43.342339 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:43.342320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlx8d" event={"ID":"84649a02-593c-4194-905b-14bada246537","Type":"ContainerStarted","Data":"7b9dc4c9dfb3a67aa3df4c3d70e13fccc27d16b3b5ddf3d10cf86e0f4239fab2"}
Jul 06 15:19:43.363377 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:43.363328    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" podStartSLOduration=162.363309565 podStartE2EDuration="2m42.363309565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:43.362932871 +0000 UTC m=+164.143361730" watchObservedRunningTime="2026-07-06 15:19:43.363309565 +0000 UTC m=+164.143738367"
Jul 06 15:19:45.350362 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.350333    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlx8d" event={"ID":"84649a02-593c-4194-905b-14bada246537","Type":"ContainerStarted","Data":"cafc69611bf32e1312f75d0ec6382930d20e6c85827b34555936bca50097accd"}
Jul 06 15:19:45.350753 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.350383    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-vlx8d" event={"ID":"84649a02-593c-4194-905b-14bada246537","Type":"ContainerStarted","Data":"bfd4518358e5fc22c75e60a63bc4f2f7fdfa02bde7881ffcbbbd53f3f3bfbc2a"}
Jul 06 15:19:45.350753 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.350430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:45.351694 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.351673    2566 generic.go:356] "Generic (PLEG): container finished" podID="955cc9f3-7e5f-4c19-a572-beb2761d9f66" containerID="942ee3ce3a79225f3a87aa61a1b62c5e54f6416181848bdab2a58757e97f1466" exitCode=255
Jul 06 15:19:45.351760 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.351711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9" event={"ID":"955cc9f3-7e5f-4c19-a572-beb2761d9f66","Type":"ContainerDied","Data":"942ee3ce3a79225f3a87aa61a1b62c5e54f6416181848bdab2a58757e97f1466"}
Jul 06 15:19:45.351961 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.351950    2566 scope.go:117] "RemoveContainer" containerID="942ee3ce3a79225f3a87aa61a1b62c5e54f6416181848bdab2a58757e97f1466"
Jul 06 15:19:45.368720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.368677    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-vlx8d" podStartSLOduration=129.383200178 podStartE2EDuration="2m11.368661553s" totalImagesPullingTime="1.985461375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:45.368024754 +0000 UTC m=+166.148453576" watchObservedRunningTime="2026-07-06 15:19:45.368661553 +0000 UTC m=+166.149090354"
Jul 06 15:19:45.964334 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:45.964281    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" podUID="b58ab71e-82eb-4a12-9b81-27d326fee966" containerName="acm-agent" probeResult="failure" output="Get \"http://10.132.0.8:8000/readyz\": dial tcp 10.132.0.8:8000: connect: connection refused"
Jul 06 15:19:46.355907 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:46.355864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6898f4959b-74rh9" event={"ID":"955cc9f3-7e5f-4c19-a572-beb2761d9f66","Type":"ContainerStarted","Data":"94b8caa0747cdc57534ce1324e34ad35ecc0d9393d250d2c0e0c7d34e786cebd"}
Jul 06 15:19:46.357072 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:46.357049    2566 generic.go:356] "Generic (PLEG): container finished" podID="b58ab71e-82eb-4a12-9b81-27d326fee966" containerID="4492c0c89827cea8482e1198c5ee810d1b1c598c3e0cc8d814f11e6240b1236d" exitCode=1
Jul 06 15:19:46.357183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:46.357120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" event={"ID":"b58ab71e-82eb-4a12-9b81-27d326fee966","Type":"ContainerDied","Data":"4492c0c89827cea8482e1198c5ee810d1b1c598c3e0cc8d814f11e6240b1236d"}
Jul 06 15:19:46.357525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:46.357510    2566 scope.go:117] "RemoveContainer" containerID="4492c0c89827cea8482e1198c5ee810d1b1c598c3e0cc8d814f11e6240b1236d"
Jul 06 15:19:47.361762 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.361722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l" event={"ID":"b58ab71e-82eb-4a12-9b81-27d326fee966","Type":"ContainerStarted","Data":"344d51b5385594fee51f66baabbad6f5db0fe72ff84a022dfc4837c83976e27a"}
Jul 06 15:19:47.362161 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.362046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:19:47.362589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.362571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7dbcffdbf7-x2d9l"
Jul 06 15:19:47.614225 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.614142    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qrbfp"]
Jul 06 15:19:47.616097 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.616080    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.618640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.618586    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 06 15:19:47.618828 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.618651    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 06 15:19:47.618828 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.618732    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 06 15:19:47.618974 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.618953    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 06 15:19:47.619041 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.619021    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 06 15:19:47.619159 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.619070    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 06 15:19:47.619159 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.619087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-v7t7r\""
Jul 06 15:19:47.711957 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.711922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-metrics-client-ca\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.711957 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.711959    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-wtmp\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-textfile\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-sys\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zbt\" (UniqueName: \"kubernetes.io/projected/23f68635-8369-4b16-9fd7-8582a76f15a2-kube-api-access-x2zbt\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712101    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-tls\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712175    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-root\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.712369 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.712194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812744 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-textfile\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-sys\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zbt\" (UniqueName: \"kubernetes.io/projected/23f68635-8369-4b16-9fd7-8582a76f15a2-kube-api-access-x2zbt\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-tls\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812860    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-root\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.812885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-metrics-client-ca\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812907    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-sys\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-wtmp\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.812955    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-root\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.813043    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-wtmp\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.813075    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-textfile\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813561 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.813485    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-metrics-client-ca\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.813561 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.813538    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-accelerators-collector-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.815083 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.815055    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.815248 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.815231    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/23f68635-8369-4b16-9fd7-8582a76f15a2-node-exporter-tls\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.821013 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.820992    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zbt\" (UniqueName: \"kubernetes.io/projected/23f68635-8369-4b16-9fd7-8582a76f15a2-kube-api-access-x2zbt\") pod \"node-exporter-qrbfp\" (UID: \"23f68635-8369-4b16-9fd7-8582a76f15a2\") " pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.924878 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:47.924808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qrbfp"
Jul 06 15:19:47.934673 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:19:47.934649    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f68635_8369_4b16_9fd7_8582a76f15a2.slice/crio-40454d583b40e533b34acd64dc8fb710ea3597a91de9dadb0a7115c1b75defb0 WatchSource:0}: Error finding container 40454d583b40e533b34acd64dc8fb710ea3597a91de9dadb0a7115c1b75defb0: Status 404 returned error can't find the container with id 40454d583b40e533b34acd64dc8fb710ea3597a91de9dadb0a7115c1b75defb0
Jul 06 15:19:48.366945 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:48.366910    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrbfp" event={"ID":"23f68635-8369-4b16-9fd7-8582a76f15a2","Type":"ContainerStarted","Data":"40454d583b40e533b34acd64dc8fb710ea3597a91de9dadb0a7115c1b75defb0"}
Jul 06 15:19:49.371087 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:49.371053    2566 generic.go:356] "Generic (PLEG): container finished" podID="23f68635-8369-4b16-9fd7-8582a76f15a2" containerID="e8f16498595d1c2a6a3e439b91570570ccbd64bc9163a5a2174d3a21487dfb4a" exitCode=0
Jul 06 15:19:49.371505 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:49.371137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrbfp" event={"ID":"23f68635-8369-4b16-9fd7-8582a76f15a2","Type":"ContainerDied","Data":"e8f16498595d1c2a6a3e439b91570570ccbd64bc9163a5a2174d3a21487dfb4a"}
Jul 06 15:19:49.717220 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:49.717138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:19:50.376413 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:50.376371    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrbfp" event={"ID":"23f68635-8369-4b16-9fd7-8582a76f15a2","Type":"ContainerStarted","Data":"1365aa6499c0ef6365757ad08a4ab817be9ba106ac057e0f6dfbd0bd43061494"}
Jul 06 15:19:50.376765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:50.376471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qrbfp" event={"ID":"23f68635-8369-4b16-9fd7-8582a76f15a2","Type":"ContainerStarted","Data":"ea7e9ff89ec7a4039fb423a7ea572f6de0c6494e8972e81102722126a6d37698"}
Jul 06 15:19:50.403625 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:50.403576    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qrbfp" podStartSLOduration=2.582298619 podStartE2EDuration="3.403560737s" totalImagesPullingTime="757.323392ms" totalInitContainerRuntime="63.938726ms" isStatefulPod=true podCreationTimestamp="2026-07-06 15:19:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:19:50.401835548 +0000 UTC m=+171.182264362" watchObservedRunningTime="2026-07-06 15:19:50.403560737 +0000 UTC m=+171.183989549"
Jul 06 15:19:55.359329 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:55.359257    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-vlx8d"
Jul 06 15:19:58.334614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:58.334574    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:19:58.338936 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:19:58.338912    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:20:23.354170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.354108    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" containerName="registry" containerID="cri-o://04e0522edd54e4cffbc99d925400b3366c987fdebf1f685f9a6de93170cddd9a" gracePeriod=30
Jul 06 15:20:23.483875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.483847    2566 generic.go:356] "Generic (PLEG): container finished" podID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" containerID="04e0522edd54e4cffbc99d925400b3366c987fdebf1f685f9a6de93170cddd9a" exitCode=0
Jul 06 15:20:23.484020 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.483889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" event={"ID":"2edfb766-da3c-4db0-8791-81bbe22dfc3f","Type":"ContainerDied","Data":"04e0522edd54e4cffbc99d925400b3366c987fdebf1f685f9a6de93170cddd9a"}
Jul 06 15:20:23.589706 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.589683    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:20:23.683528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683513    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683536    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tfg4q\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683554    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683630    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.683801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.683659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration\") pod \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\" (UID: \"2edfb766-da3c-4db0-8791-81bbe22dfc3f\") "
Jul 06 15:20:23.684108 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.684059    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:20:23.684299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.684274    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:20:23.686025 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.685993    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:20:23.686332 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.686306    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:20:23.686648 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.686614    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q" (OuterVolumeSpecName: "kube-api-access-tfg4q") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "kube-api-access-tfg4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:20:23.686648 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.686629    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:20:23.686811 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.686702    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:20:23.692303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.692272    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2edfb766-da3c-4db0-8791-81bbe22dfc3f" (UID: "2edfb766-da3c-4db0-8791-81bbe22dfc3f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:20:23.784442 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784412    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-tls\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784442 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784438    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-installation-pull-secrets\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784442 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784449    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfg4q\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-kube-api-access-tfg4q\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784458    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-registry-certificates\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784467    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2edfb766-da3c-4db0-8791-81bbe22dfc3f-ca-trust-extracted\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784475    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2edfb766-da3c-4db0-8791-81bbe22dfc3f-bound-sa-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784484    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2edfb766-da3c-4db0-8791-81bbe22dfc3f-image-registry-private-configuration\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:23.784642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:23.784492    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2edfb766-da3c-4db0-8791-81bbe22dfc3f-trusted-ca\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:20:24.487948 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:24.487898    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6" event={"ID":"2edfb766-da3c-4db0-8791-81bbe22dfc3f","Type":"ContainerDied","Data":"128fea2608743b54c0e64039a83e6505f62665006bdb7ebe90d1a8bf2f468f09"}
Jul 06 15:20:24.487948 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:24.487948    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-767857c9d5-7vfj6"
Jul 06 15:20:24.488444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:24.487989    2566 scope.go:117] "RemoveContainer" containerID="04e0522edd54e4cffbc99d925400b3366c987fdebf1f685f9a6de93170cddd9a"
Jul 06 15:20:24.509824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:24.509799    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:20:24.516232 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:24.516210    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-767857c9d5-7vfj6"]
Jul 06 15:20:25.720030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:25.719994    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" path="/var/lib/kubelet/pods/2edfb766-da3c-4db0-8791-81bbe22dfc3f/volumes"
Jul 06 15:20:34.926729 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:34.926691    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" podUID="6d537d6b-7607-4d25-b316-d98f263290cb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 15:20:44.925978 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:44.925941    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" podUID="6d537d6b-7607-4d25-b316-d98f263290cb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 15:20:54.925933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:54.925895    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" podUID="6d537d6b-7607-4d25-b316-d98f263290cb" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 06 15:20:54.926430 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:54.925975    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg"
Jul 06 15:20:54.926476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:54.926435    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"33a290666b98531e971d9ab5dadee953030de90ed55b438f2cdd5c451d10e278"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 06 15:20:54.926512 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:54.926480    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" podUID="6d537d6b-7607-4d25-b316-d98f263290cb" containerName="service-proxy" containerID="cri-o://33a290666b98531e971d9ab5dadee953030de90ed55b438f2cdd5c451d10e278" gracePeriod=30
Jul 06 15:20:55.586815 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:55.586781    2566 generic.go:356] "Generic (PLEG): container finished" podID="6d537d6b-7607-4d25-b316-d98f263290cb" containerID="33a290666b98531e971d9ab5dadee953030de90ed55b438f2cdd5c451d10e278" exitCode=2
Jul 06 15:20:55.586983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:55.586830    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerDied","Data":"33a290666b98531e971d9ab5dadee953030de90ed55b438f2cdd5c451d10e278"}
Jul 06 15:20:55.586983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:20:55.586868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-699b8646fb-xsfxg" event={"ID":"6d537d6b-7607-4d25-b316-d98f263290cb","Type":"ContainerStarted","Data":"5d88c82e6258b6534c29e4c1ec18169c341fb1e64c9934eb0a5ddebdf962cb7a"}
Jul 06 15:21:11.532497 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:11.532453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:21:11.534727 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:11.534705    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b57e3d86-626d-4705-b410-5694cfd1accd-metrics-certs\") pod \"network-metrics-daemon-7rrbh\" (UID: \"b57e3d86-626d-4705-b410-5694cfd1accd\") " pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:21:11.621182 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:11.621152    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gh99x\""
Jul 06 15:21:11.628012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:11.627989    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7rrbh"
Jul 06 15:21:11.758464 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:11.758432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7rrbh"]
Jul 06 15:21:11.761582 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:21:11.761560    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57e3d86_626d_4705_b410_5694cfd1accd.slice/crio-6e1b092aae903b2e55874346afe823d1a9bc1fcad554dceb22bec86f703e4992 WatchSource:0}: Error finding container 6e1b092aae903b2e55874346afe823d1a9bc1fcad554dceb22bec86f703e4992: Status 404 returned error can't find the container with id 6e1b092aae903b2e55874346afe823d1a9bc1fcad554dceb22bec86f703e4992
Jul 06 15:21:12.642745 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:12.642712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rrbh" event={"ID":"b57e3d86-626d-4705-b410-5694cfd1accd","Type":"ContainerStarted","Data":"6e1b092aae903b2e55874346afe823d1a9bc1fcad554dceb22bec86f703e4992"}
Jul 06 15:21:13.647082 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:13.647046    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rrbh" event={"ID":"b57e3d86-626d-4705-b410-5694cfd1accd","Type":"ContainerStarted","Data":"a4ee4a51da1adffe726fd05d1517d914670c5dad614f8d9b2ba11004285c25e0"}
Jul 06 15:21:13.647592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:13.647105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7rrbh" event={"ID":"b57e3d86-626d-4705-b410-5694cfd1accd","Type":"ContainerStarted","Data":"0b53374e75aa6d080b37112f1fcbec63862369f1e1d219aef7288ebf817150e7"}
Jul 06 15:21:13.670116 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:13.670070    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7rrbh" podStartSLOduration=252.3601403 podStartE2EDuration="4m13.670056104s" totalImagesPullingTime="1.309915804s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:21:13.668806086 +0000 UTC m=+254.449234905" watchObservedRunningTime="2026-07-06 15:21:13.670056104 +0000 UTC m=+254.450484905"
Jul 06 15:21:59.623680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:21:59.623650    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 06 15:22:50.514661 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.514626    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vk7cf"]
Jul 06 15:22:50.515081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.514870    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" containerName="registry"
Jul 06 15:22:50.515081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.514881    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" containerName="registry"
Jul 06 15:22:50.515081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.514925    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2edfb766-da3c-4db0-8791-81bbe22dfc3f" containerName="registry"
Jul 06 15:22:50.517529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.517514    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.520185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.520157    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-chslp\""
Jul 06 15:22:50.520185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.520177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 06 15:22:50.520348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.520177    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 06 15:22:50.529134 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.529114    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vk7cf"]
Jul 06 15:22:50.612471 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.612435    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.612471 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.612481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zf7p\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-kube-api-access-4zf7p\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.713283 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.713237    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zf7p\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-kube-api-access-4zf7p\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.713449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.713311    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.726678 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.726645    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-bound-sa-token\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.726869 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.726849    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zf7p\" (UniqueName: \"kubernetes.io/projected/42729c55-f345-4a32-81e1-5389bec392c7-kube-api-access-4zf7p\") pod \"cert-manager-55b4d954d5-vk7cf\" (UID: \"42729c55-f345-4a32-81e1-5389bec392c7\") " pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.826075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.825993    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-vk7cf"
Jul 06 15:22:50.953789 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.953759    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-vk7cf"]
Jul 06 15:22:50.956820 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:22:50.956793    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42729c55_f345_4a32_81e1_5389bec392c7.slice/crio-5390dfd30023c52436980dbea7aab9cd559cf886be9221d88177f60b3886e23c WatchSource:0}: Error finding container 5390dfd30023c52436980dbea7aab9cd559cf886be9221d88177f60b3886e23c: Status 404 returned error can't find the container with id 5390dfd30023c52436980dbea7aab9cd559cf886be9221d88177f60b3886e23c
Jul 06 15:22:50.958773 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.958758    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:22:50.961416 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:50.961372    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vk7cf" event={"ID":"42729c55-f345-4a32-81e1-5389bec392c7","Type":"ContainerStarted","Data":"5390dfd30023c52436980dbea7aab9cd559cf886be9221d88177f60b3886e23c"}
Jul 06 15:22:54.976905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:54.976871    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vk7cf" event={"ID":"42729c55-f345-4a32-81e1-5389bec392c7","Type":"ContainerStarted","Data":"383b5b5029fe94c6049d76f62cace32f07d6fd883541373cc4af3149038f2c7e"}
Jul 06 15:22:54.997551 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:22:54.997495    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-vk7cf" podStartSLOduration=2.008674355 podStartE2EDuration="4.997479333s" totalImagesPullingTime="2.988804978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:22:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:22:54.996058702 +0000 UTC m=+355.776487524" watchObservedRunningTime="2026-07-06 15:22:54.997479333 +0000 UTC m=+355.777908134"
Jul 06 15:23:17.697226 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.697191    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"]
Jul 06 15:23:17.748371 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.748339    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"]
Jul 06 15:23:17.748565 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.748511    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:17.754374 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.754350    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 06 15:23:17.754577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.754350    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 06 15:23:17.754577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.754356    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 06 15:23:17.754712 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.754696    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 06 15:23:17.755159 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.754763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8n8jq\""
Jul 06 15:23:17.907686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.907642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:17.907857 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.907748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:17.907857 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.907794    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn7wl\" (UniqueName: \"kubernetes.io/projected/6ced8269-b9bb-47c9-b274-68a24ef89081-kube-api-access-fn7wl\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:17.907857 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.907825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:17.908068 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:17.907865    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008145    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008187    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn7wl\" (UniqueName: \"kubernetes.io/projected/6ced8269-b9bb-47c9-b274-68a24ef89081-kube-api-access-fn7wl\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008211    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008257    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.008588 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.008565    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-manifests\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.010687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.010668    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.010855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.010833    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6ced8269-b9bb-47c9-b274-68a24ef89081-webhook-cert\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.011717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.011702    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6ced8269-b9bb-47c9-b274-68a24ef89081-tmp\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.018940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.018912    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn7wl\" (UniqueName: \"kubernetes.io/projected/6ced8269-b9bb-47c9-b274-68a24ef89081-kube-api-access-fn7wl\") pod \"opendatahub-operator-controller-manager-9957545c6-kt6bg\" (UID: \"6ced8269-b9bb-47c9-b274-68a24ef89081\") " pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.059026 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.058981    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:18.196117 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:18.196078    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"]
Jul 06 15:23:18.199305 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:23:18.199278    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ced8269_b9bb_47c9_b274_68a24ef89081.slice/crio-936b80809068f29ccfd907f27112860319bd282682f7ef57d448b27ee28e86fb WatchSource:0}: Error finding container 936b80809068f29ccfd907f27112860319bd282682f7ef57d448b27ee28e86fb: Status 404 returned error can't find the container with id 936b80809068f29ccfd907f27112860319bd282682f7ef57d448b27ee28e86fb
Jul 06 15:23:19.056468 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:19.056431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg" event={"ID":"6ced8269-b9bb-47c9-b274-68a24ef89081","Type":"ContainerStarted","Data":"936b80809068f29ccfd907f27112860319bd282682f7ef57d448b27ee28e86fb"}
Jul 06 15:23:22.068962 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:22.068876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg" event={"ID":"6ced8269-b9bb-47c9-b274-68a24ef89081","Type":"ContainerStarted","Data":"f6fb1ddc4f91077dd576487cf351f147f168349d55ec361aaa192c4165028297"}
Jul 06 15:23:23.073961 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:23.073930    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ced8269-b9bb-47c9-b274-68a24ef89081" containerID="f6fb1ddc4f91077dd576487cf351f147f168349d55ec361aaa192c4165028297" exitCode=0
Jul 06 15:23:23.074328 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:23.074005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg" event={"ID":"6ced8269-b9bb-47c9-b274-68a24ef89081","Type":"ContainerDied","Data":"f6fb1ddc4f91077dd576487cf351f147f168349d55ec361aaa192c4165028297"}
Jul 06 15:23:24.080012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:24.079981    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg" event={"ID":"6ced8269-b9bb-47c9-b274-68a24ef89081","Type":"ContainerStarted","Data":"61683c42c75a0efef3fd97009f80891c6f853603ba3d30ea5914d7a3dff6955e"}
Jul 06 15:23:24.080381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:24.080106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:24.130313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:24.130258    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg" podStartSLOduration=2.132354225 podStartE2EDuration="7.13024102s" totalImagesPullingTime="3.930425212s" totalInitContainerRuntime="1.067461583s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:24.127698113 +0000 UTC m=+384.908126927" watchObservedRunningTime="2026-07-06 15:23:24.13024102 +0000 UTC m=+384.910669820"
Jul 06 15:23:35.086476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:35.086446    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9957545c6-kt6bg"
Jul 06 15:23:38.608201 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.608161    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"]
Jul 06 15:23:38.616523 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.616496    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.619941 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.619917    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 06 15:23:38.620250 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.620230    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-dkl65\""
Jul 06 15:23:38.620446 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.620430    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 06 15:23:38.620517 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.620488    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 06 15:23:38.620572 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.620523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 06 15:23:38.635468 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.635441    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"]
Jul 06 15:23:38.764713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.764668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xb77\" (UniqueName: \"kubernetes.io/projected/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-kube-api-access-2xb77\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.764713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.764714    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tmp\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.764937 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.764744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tls-certs\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.782625 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.782587    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gt8sw"]
Jul 06 15:23:38.785642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.785618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:38.805840 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.805811    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2xmcc\""
Jul 06 15:23:38.815447 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.815422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 06 15:23:38.843984 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.843941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gt8sw"]
Jul 06 15:23:38.866039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.865954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xb77\" (UniqueName: \"kubernetes.io/projected/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-kube-api-access-2xb77\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.866039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.865992    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tmp\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.866039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.866026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:38.866252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.866144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tls-certs\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.866252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.866212    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpm6h\" (UniqueName: \"kubernetes.io/projected/f0dc9e2e-e399-484e-ad59-d0749c967805-kube-api-access-kpm6h\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:38.868260 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.868237    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tmp\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.868500 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.868481    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-tls-certs\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.917287 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.917248    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xb77\" (UniqueName: \"kubernetes.io/projected/e54f8ea3-0cfb-49bc-a045-810ef7a004b5-kube-api-access-2xb77\") pod \"kube-auth-proxy-577f8bd98b-jm7cf\" (UID: \"e54f8ea3-0cfb-49bc-a045-810ef7a004b5\") " pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.926070 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.926040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"
Jul 06 15:23:38.967526 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.967488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:38.967706 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.967573    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpm6h\" (UniqueName: \"kubernetes.io/projected/f0dc9e2e-e399-484e-ad59-d0749c967805-kube-api-access-kpm6h\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:38.967706 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:38.967689    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 06 15:23:38.967800 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:38.967787    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert podName:f0dc9e2e-e399-484e-ad59-d0749c967805 nodeName:}" failed. No retries permitted until 2026-07-06 15:23:39.467763162 +0000 UTC m=+400.248191959 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert") pod "odh-model-controller-7998c67dc4-gt8sw" (UID: "f0dc9e2e-e399-484e-ad59-d0749c967805") : secret "odh-model-controller-webhook-cert" not found
Jul 06 15:23:38.984183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:38.984160    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpm6h\" (UniqueName: \"kubernetes.io/projected/f0dc9e2e-e399-484e-ad59-d0749c967805-kube-api-access-kpm6h\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:39.087697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.087665    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf"]
Jul 06 15:23:39.093012 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:23:39.092983    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode54f8ea3_0cfb_49bc_a045_810ef7a004b5.slice/crio-5c6e3bcf1190cca6d0ad0fe1e8e83a84b28fa8c7a5090af839962cc297cd73c5 WatchSource:0}: Error finding container 5c6e3bcf1190cca6d0ad0fe1e8e83a84b28fa8c7a5090af839962cc297cd73c5: Status 404 returned error can't find the container with id 5c6e3bcf1190cca6d0ad0fe1e8e83a84b28fa8c7a5090af839962cc297cd73c5
Jul 06 15:23:39.131951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.131865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf" event={"ID":"e54f8ea3-0cfb-49bc-a045-810ef7a004b5","Type":"ContainerStarted","Data":"5c6e3bcf1190cca6d0ad0fe1e8e83a84b28fa8c7a5090af839962cc297cd73c5"}
Jul 06 15:23:39.476492 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.472143    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:39.476492 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.475568    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f0dc9e2e-e399-484e-ad59-d0749c967805-cert\") pod \"odh-model-controller-7998c67dc4-gt8sw\" (UID: \"f0dc9e2e-e399-484e-ad59-d0749c967805\") " pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:39.695869 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.695824    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:39.860165 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:39.860132    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-gt8sw"]
Jul 06 15:23:39.862755 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:23:39.862724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0dc9e2e_e399_484e_ad59_d0749c967805.slice/crio-75db9cd6b963984ca151115b2fd9694ba500c73b4aaa7b3d83cac45f9e133afe WatchSource:0}: Error finding container 75db9cd6b963984ca151115b2fd9694ba500c73b4aaa7b3d83cac45f9e133afe: Status 404 returned error can't find the container with id 75db9cd6b963984ca151115b2fd9694ba500c73b4aaa7b3d83cac45f9e133afe
Jul 06 15:23:40.137250 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.137208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw" event={"ID":"f0dc9e2e-e399-484e-ad59-d0749c967805","Type":"ContainerStarted","Data":"75db9cd6b963984ca151115b2fd9694ba500c73b4aaa7b3d83cac45f9e133afe"}
Jul 06 15:23:40.380052 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.380008    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:40.383181 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.383160    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.386438 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.386413    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 06 15:23:40.386618 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.386422    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 06 15:23:40.386860 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.386843    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 06 15:23:40.386942 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.386931    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 15:23:40.386997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.386941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kl8p8\""
Jul 06 15:23:40.407018 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.406977    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:40.481926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.481885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.481946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpngv\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.482011    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.482064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.482096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.482118    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.482140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.482141    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582633 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582597    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.582875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.583128 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.582927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpngv\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.583595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.583449    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.586161 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.586106    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.586468 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.586406    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.586884 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.586860    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.587978 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.587942    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.592956 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.592906    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.594086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.594059    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpngv\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv\") pod \"istiod-openshift-gateway-7574459656-rd4gm\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.694100 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.694015    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:40.861342 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:40.861310    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:40.861781 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:23:40.861757    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26b153c6_4a9c_426f_86ca_cd36a80623bd.slice/crio-f78a2fcbedcbb72a7ca566d51cd036671daf6bffb517bd8d25468db6f6168bd3 WatchSource:0}: Error finding container f78a2fcbedcbb72a7ca566d51cd036671daf6bffb517bd8d25468db6f6168bd3: Status 404 returned error can't find the container with id f78a2fcbedcbb72a7ca566d51cd036671daf6bffb517bd8d25468db6f6168bd3
Jul 06 15:23:41.142680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:41.142642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" event={"ID":"26b153c6-4a9c-426f-86ca-cd36a80623bd","Type":"ContainerStarted","Data":"f78a2fcbedcbb72a7ca566d51cd036671daf6bffb517bd8d25468db6f6168bd3"}
Jul 06 15:23:45.219417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:45.219370    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:23:45.219702 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:45.219451    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:23:46.165351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.165313    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" event={"ID":"26b153c6-4a9c-426f-86ca-cd36a80623bd","Type":"ContainerStarted","Data":"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"}
Jul 06 15:23:46.165545 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.165445    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:46.167529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.167484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf" event={"ID":"e54f8ea3-0cfb-49bc-a045-810ef7a004b5","Type":"ContainerStarted","Data":"8d1b3adae791011020302b2cba2c937fea37630cb0959e0c1c0b833be21fc78c"}
Jul 06 15:23:46.167701 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.167665    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rd4gm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:23:46.167800 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.167725    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:23:46.171008 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.170976    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw" event={"ID":"f0dc9e2e-e399-484e-ad59-d0749c967805","Type":"ContainerStarted","Data":"e15734daf2ae10d8791a2618be32c36f8a99a6fe025a8e6265017732298ddc26"}
Jul 06 15:23:46.175408 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.171884    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:46.214509 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.214454    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" podStartSLOduration=1.859258675 podStartE2EDuration="6.214439167s" totalImagesPullingTime="4.355180492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:46.212986564 +0000 UTC m=+406.993415365" watchObservedRunningTime="2026-07-06 15:23:46.214439167 +0000 UTC m=+406.994867967"
Jul 06 15:23:46.253155 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.253101    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw" podStartSLOduration=2.898451047 podStartE2EDuration="8.253084843s" totalImagesPullingTime="5.354633796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:46.251514528 +0000 UTC m=+407.031943328" watchObservedRunningTime="2026-07-06 15:23:46.253084843 +0000 UTC m=+407.033513643"
Jul 06 15:23:46.291885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:46.291820    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-577f8bd98b-jm7cf" podStartSLOduration=2.173318988 podStartE2EDuration="8.29180023s" totalImagesPullingTime="6.118481242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:46.289882323 +0000 UTC m=+407.070311124" watchObservedRunningTime="2026-07-06 15:23:46.29180023 +0000 UTC m=+407.072229032"
Jul 06 15:23:47.177977 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:47.177946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:51.422804 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.422770    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"]
Jul 06 15:23:51.426289 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.426274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:51.430776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.430755    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 06 15:23:51.430892 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.430794    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lp6vn\""
Jul 06 15:23:51.453138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.453110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"]
Jul 06 15:23:51.474962 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.474922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:51.475115 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.475030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrpb\" (UniqueName: \"kubernetes.io/projected/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-kube-api-access-gkrpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:51.575381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.575342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrpb\" (UniqueName: \"kubernetes.io/projected/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-kube-api-access-gkrpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:51.575583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.575412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:51.575583 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:51.575548    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 06 15:23:51.575656 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:51.575619    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert podName:d6f4dab9-f7df-4694-9ae3-8f3a2973c218 nodeName:}" failed. No retries permitted until 2026-07-06 15:23:52.075601007 +0000 UTC m=+412.856029788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert") pod "llmisvc-controller-manager-777c6bd7d5-gsq8j" (UID: "d6f4dab9-f7df-4694-9ae3-8f3a2973c218") : secret "llmisvc-webhook-server-cert" not found
Jul 06 15:23:51.601609 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:51.601581    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrpb\" (UniqueName: \"kubernetes.io/projected/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-kube-api-access-gkrpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:52.079175 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:52.079142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:52.079358 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:52.079334    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 06 15:23:52.079464 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:52.079452    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert podName:d6f4dab9-f7df-4694-9ae3-8f3a2973c218 nodeName:}" failed. No retries permitted until 2026-07-06 15:23:53.079430982 +0000 UTC m=+413.859859776 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert") pod "llmisvc-controller-manager-777c6bd7d5-gsq8j" (UID: "d6f4dab9-f7df-4694-9ae3-8f3a2973c218") : secret "llmisvc-webhook-server-cert" not found
Jul 06 15:23:53.085985 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.085947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:53.088334 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.088305    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6f4dab9-f7df-4694-9ae3-8f3a2973c218-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gsq8j\" (UID: \"d6f4dab9-f7df-4694-9ae3-8f3a2973c218\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:53.161012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.160980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:53.161178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.161157    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery" containerID="cri-o://ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe" gracePeriod=30
Jul 06 15:23:53.177686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.177657    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-rd4gm container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.17:8080/ready\": dial tcp 10.132.0.17:8080: connect: connection refused" start-of-body=
Jul 06 15:23:53.177836 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.177709    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.17:8080/ready\": dial tcp 10.132.0.17:8080: connect: connection refused"
Jul 06 15:23:53.236451 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.236409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:53.411562 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.411537    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:53.489861 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.489826    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490035 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.489877    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490035 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.489910    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490035 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.489937    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490035 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.489963    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490035 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.490006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xpngv\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490287 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.490039    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig\") pod \"26b153c6-4a9c-426f-86ca-cd36a80623bd\" (UID: \"26b153c6-4a9c-426f-86ca-cd36a80623bd\") "
Jul 06 15:23:53.490344 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.490323    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:23:53.492335 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492306    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs" (OuterVolumeSpecName: "local-certs") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:23:53.492548 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492336    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts" (OuterVolumeSpecName: "cacerts") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:23:53.492609 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492576    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:23:53.492648 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492621    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:23:53.492648 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492630    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv" (OuterVolumeSpecName: "kube-api-access-xpngv") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "kube-api-access-xpngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:23:53.492705 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.492643    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "26b153c6-4a9c-426f-86ca-cd36a80623bd" (UID: "26b153c6-4a9c-426f-86ca-cd36a80623bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:23:53.591066 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591032    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpngv\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-kube-api-access-xpngv\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591066 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591058    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-kubeconfig\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591066 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591067    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-ca-configmap\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591077    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-cacerts\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591086    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/26b153c6-4a9c-426f-86ca-cd36a80623bd-local-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591094    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-csr-dns-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.591290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.591101    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26b153c6-4a9c-426f-86ca-cd36a80623bd-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:23:53.598827 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:53.598792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"]
Jul 06 15:23:53.602430 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:23:53.602385    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd6f4dab9_f7df_4694_9ae3_8f3a2973c218.slice/crio-016d0320e9d7a3836dc9853eb575ee7c0830287ffbc2e51cf598d9a1f74fa1a3 WatchSource:0}: Error finding container 016d0320e9d7a3836dc9853eb575ee7c0830287ffbc2e51cf598d9a1f74fa1a3: Status 404 returned error can't find the container with id 016d0320e9d7a3836dc9853eb575ee7c0830287ffbc2e51cf598d9a1f74fa1a3
Jul 06 15:23:54.204341 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.204307    2566 generic.go:356] "Generic (PLEG): container finished" podID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerID="ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe" exitCode=0
Jul 06 15:23:54.204758 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.204348    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"
Jul 06 15:23:54.204758 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.204409    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" event={"ID":"26b153c6-4a9c-426f-86ca-cd36a80623bd","Type":"ContainerDied","Data":"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"}
Jul 06 15:23:54.204758 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.204466    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm" event={"ID":"26b153c6-4a9c-426f-86ca-cd36a80623bd","Type":"ContainerDied","Data":"f78a2fcbedcbb72a7ca566d51cd036671daf6bffb517bd8d25468db6f6168bd3"}
Jul 06 15:23:54.204758 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.204496    2566 scope.go:117] "RemoveContainer" containerID="ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"
Jul 06 15:23:54.205966 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.205798    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j" event={"ID":"d6f4dab9-f7df-4694-9ae3-8f3a2973c218","Type":"ContainerStarted","Data":"016d0320e9d7a3836dc9853eb575ee7c0830287ffbc2e51cf598d9a1f74fa1a3"}
Jul 06 15:23:54.212680 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.212654    2566 scope.go:117] "RemoveContainer" containerID="ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"
Jul 06 15:23:54.213002 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:23:54.212974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe\": container with ID starting with ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe not found: ID does not exist" containerID="ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"
Jul 06 15:23:54.213047 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.213012    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe"} err="failed to get container status \"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe\": rpc error: code = NotFound desc = could not find container \"ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe\": container with ID starting with ebf7f06610e9298cffd02f0dc0475acdaa1f9debb7c024623de61a16a133ecbe not found: ID does not exist"
Jul 06 15:23:54.248840 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.248808    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:54.256274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:54.255652    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-rd4gm"]
Jul 06 15:23:55.720509 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:55.720474    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" path="/var/lib/kubelet/pods/26b153c6-4a9c-426f-86ca-cd36a80623bd/volumes"
Jul 06 15:23:57.178814 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:57.178786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-gt8sw"
Jul 06 15:23:57.219675 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:57.219636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j" event={"ID":"d6f4dab9-f7df-4694-9ae3-8f3a2973c218","Type":"ContainerStarted","Data":"2faa804fc4f280b5d5ab8433addd06f203847839c6981b10f9e94803aa1d520d"}
Jul 06 15:23:57.219874 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:57.219729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:23:57.276467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:23:57.276406    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j" podStartSLOduration=3.09057809 podStartE2EDuration="6.276369632s" totalImagesPullingTime="3.185791542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:23:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:23:57.27482406 +0000 UTC m=+418.055252861" watchObservedRunningTime="2026-07-06 15:23:57.276369632 +0000 UTC m=+418.056798434"
Jul 06 15:24:28.225890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:28.225818    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gsq8j"
Jul 06 15:24:39.508958 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.508918    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:39.509460 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.509295    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery"
Jul 06 15:24:39.509460 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.509313    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery"
Jul 06 15:24:39.509460 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.509406    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="26b153c6-4a9c-426f-86ca-cd36a80623bd" containerName="discovery"
Jul 06 15:24:39.517473 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.517447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:39.520335 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.520311    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-rfgj7\""
Jul 06 15:24:39.520621 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.520597    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 06 15:24:39.521034 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.521020    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 06 15:24:39.527806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.527774    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:39.641530 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.641495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hk44\" (UniqueName: \"kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44\") pod \"kuadrant-operator-catalog-zhndm\" (UID: \"44427d3f-3494-4fcd-8e5f-d339a6341f94\") " pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:39.742535 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.742494    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hk44\" (UniqueName: \"kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44\") pod \"kuadrant-operator-catalog-zhndm\" (UID: \"44427d3f-3494-4fcd-8e5f-d339a6341f94\") " pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:39.751511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.751484    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hk44\" (UniqueName: \"kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44\") pod \"kuadrant-operator-catalog-zhndm\" (UID: \"44427d3f-3494-4fcd-8e5f-d339a6341f94\") " pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:39.779412 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.779324    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:39.779586 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.779564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:39.926807 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.926671    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:39.928900 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:39.928874    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44427d3f_3494_4fcd_8e5f_d339a6341f94.slice/crio-277efefcdefa7edc4d901603acdf0de4cf649fe88eb9a1f3440d11484491f34c WatchSource:0}: Error finding container 277efefcdefa7edc4d901603acdf0de4cf649fe88eb9a1f3440d11484491f34c: Status 404 returned error can't find the container with id 277efefcdefa7edc4d901603acdf0de4cf649fe88eb9a1f3440d11484491f34c
Jul 06 15:24:39.987280 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.987256    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gs97p"]
Jul 06 15:24:39.992129 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:39.992115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:40.002805 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.002778    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gs97p"]
Jul 06 15:24:40.147470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.147439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkd9\" (UniqueName: \"kubernetes.io/projected/7c5d7737-644e-4728-b383-8749491d1256-kube-api-access-nrkd9\") pod \"kuadrant-operator-catalog-gs97p\" (UID: \"7c5d7737-644e-4728-b383-8749491d1256\") " pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:40.248211 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.248177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkd9\" (UniqueName: \"kubernetes.io/projected/7c5d7737-644e-4728-b383-8749491d1256-kube-api-access-nrkd9\") pod \"kuadrant-operator-catalog-gs97p\" (UID: \"7c5d7737-644e-4728-b383-8749491d1256\") " pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:40.258625 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.258598    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkd9\" (UniqueName: \"kubernetes.io/projected/7c5d7737-644e-4728-b383-8749491d1256-kube-api-access-nrkd9\") pod \"kuadrant-operator-catalog-gs97p\" (UID: \"7c5d7737-644e-4728-b383-8749491d1256\") " pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:40.300443 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.300411    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:40.369255 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.369215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" event={"ID":"44427d3f-3494-4fcd-8e5f-d339a6341f94","Type":"ContainerStarted","Data":"277efefcdefa7edc4d901603acdf0de4cf649fe88eb9a1f3440d11484491f34c"}
Jul 06 15:24:40.429046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:40.428975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gs97p"]
Jul 06 15:24:40.432150 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:40.432127    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5d7737_644e_4728_b383_8749491d1256.slice/crio-75463e5b898450f21e515e31d3e657f88c3eb8070cb5cf22a3b597a5ca35a3dc WatchSource:0}: Error finding container 75463e5b898450f21e515e31d3e657f88c3eb8070cb5cf22a3b597a5ca35a3dc: Status 404 returned error can't find the container with id 75463e5b898450f21e515e31d3e657f88c3eb8070cb5cf22a3b597a5ca35a3dc
Jul 06 15:24:41.374590 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:41.374555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gs97p" event={"ID":"7c5d7737-644e-4728-b383-8749491d1256","Type":"ContainerStarted","Data":"75463e5b898450f21e515e31d3e657f88c3eb8070cb5cf22a3b597a5ca35a3dc"}
Jul 06 15:24:42.381045 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.380999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gs97p" event={"ID":"7c5d7737-644e-4728-b383-8749491d1256","Type":"ContainerStarted","Data":"a4fab8841018717bd6d46b4ec58910c893b20aa78f67239301d5957114c23223"}
Jul 06 15:24:42.383904 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.383703    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" event={"ID":"44427d3f-3494-4fcd-8e5f-d339a6341f94","Type":"ContainerStarted","Data":"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"}
Jul 06 15:24:42.383904 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.383858    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" podUID="44427d3f-3494-4fcd-8e5f-d339a6341f94" containerName="registry-server" containerID="cri-o://3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f" gracePeriod=2
Jul 06 15:24:42.399011 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.398956    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gs97p" podStartSLOduration=1.565614437 podStartE2EDuration="3.398937929s" totalImagesPullingTime="1.833323492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:24:42.397988864 +0000 UTC m=+463.178417665" watchObservedRunningTime="2026-07-06 15:24:42.398937929 +0000 UTC m=+463.179366709"
Jul 06 15:24:42.415164 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.415109    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" podStartSLOduration=1.082180427 podStartE2EDuration="3.415091628s" totalImagesPullingTime="2.332911201s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:24:42.413823262 +0000 UTC m=+463.194252065" watchObservedRunningTime="2026-07-06 15:24:42.415091628 +0000 UTC m=+463.195520428"
Jul 06 15:24:42.625054 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.625034    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zhndm_44427d3f-3494-4fcd-8e5f-d339a6341f94/registry-server/0.log"
Jul 06 15:24:42.627347 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.627327    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:42.771844 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.771815    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hk44\" (UniqueName: \"kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44\") pod \"44427d3f-3494-4fcd-8e5f-d339a6341f94\" (UID: \"44427d3f-3494-4fcd-8e5f-d339a6341f94\") "
Jul 06 15:24:42.773911 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.773885    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44" (OuterVolumeSpecName: "kube-api-access-6hk44") pod "44427d3f-3494-4fcd-8e5f-d339a6341f94" (UID: "44427d3f-3494-4fcd-8e5f-d339a6341f94"). InnerVolumeSpecName "kube-api-access-6hk44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:24:42.873140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:42.873111    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hk44\" (UniqueName: \"kubernetes.io/projected/44427d3f-3494-4fcd-8e5f-d339a6341f94-kube-api-access-6hk44\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:24:43.388615 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388588    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-zhndm_44427d3f-3494-4fcd-8e5f-d339a6341f94/registry-server/0.log"
Jul 06 15:24:43.388983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388625    2566 generic.go:356] "Generic (PLEG): container finished" podID="44427d3f-3494-4fcd-8e5f-d339a6341f94" containerID="3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f" exitCode=1
Jul 06 15:24:43.388983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388673    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-zhndm"
Jul 06 15:24:43.388983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388704    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" event={"ID":"44427d3f-3494-4fcd-8e5f-d339a6341f94","Type":"ContainerDied","Data":"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"}
Jul 06 15:24:43.388983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-zhndm" event={"ID":"44427d3f-3494-4fcd-8e5f-d339a6341f94","Type":"ContainerDied","Data":"277efefcdefa7edc4d901603acdf0de4cf649fe88eb9a1f3440d11484491f34c"}
Jul 06 15:24:43.388983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.388759    2566 scope.go:117] "RemoveContainer" containerID="3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"
Jul 06 15:24:43.397432 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.397408    2566 scope.go:117] "RemoveContainer" containerID="3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"
Jul 06 15:24:43.397701 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:24:43.397683    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f\": container with ID starting with 3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f not found: ID does not exist" containerID="3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"
Jul 06 15:24:43.397768 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.397713    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f"} err="failed to get container status \"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f\": rpc error: code = NotFound desc = could not find container \"3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f\": container with ID starting with 3f67eafa94d6ea31b9a441f3d2b303fbd1d25a30bc31e16d602abe9c72376c7f not found: ID does not exist"
Jul 06 15:24:43.410042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.410017    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:43.413297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.413276    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-zhndm"]
Jul 06 15:24:43.719772 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:43.719737    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44427d3f-3494-4fcd-8e5f-d339a6341f94" path="/var/lib/kubelet/pods/44427d3f-3494-4fcd-8e5f-d339a6341f94/volumes"
Jul 06 15:24:50.300807 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:50.300770    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:50.301191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:50.300846    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:50.322674 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:50.322646    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:50.439215 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:50.439189    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-gs97p"
Jul 06 15:24:51.830833 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.830802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"]
Jul 06 15:24:51.831197 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.831095    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44427d3f-3494-4fcd-8e5f-d339a6341f94" containerName="registry-server"
Jul 06 15:24:51.831197 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.831107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="44427d3f-3494-4fcd-8e5f-d339a6341f94" containerName="registry-server"
Jul 06 15:24:51.831197 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.831160    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="44427d3f-3494-4fcd-8e5f-d339a6341f94" containerName="registry-server"
Jul 06 15:24:51.835386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.835368    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:51.837882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.837866    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xr89z\""
Jul 06 15:24:51.843755 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.843730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"]
Jul 06 15:24:51.945436 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.945381    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdjvb\" (UniqueName: \"kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:51.945436 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.945439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:51.945670 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:51.945534    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.046373 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.046339    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdjvb\" (UniqueName: \"kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.046373 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.046378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.046639 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.046436    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.046858 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.046834    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.046924 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.046863    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.056671 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.056644    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdjvb\" (UniqueName: \"kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.144383 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.144340    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:24:52.273795 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.273740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"]
Jul 06 15:24:52.276002 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:52.275973    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f50476d_c6c7_4766_bb5a_72a40ebaca96.slice/crio-03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861 WatchSource:0}: Error finding container 03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861: Status 404 returned error can't find the container with id 03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861
Jul 06 15:24:52.426464 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.426338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs" event={"ID":"9f50476d-c6c7-4766-bb5a-72a40ebaca96","Type":"ContainerStarted","Data":"03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861"}
Jul 06 15:24:52.630304 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.630265    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"]
Jul 06 15:24:52.634721 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.634696    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.644223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.644198    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"]
Jul 06 15:24:52.752566 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.752482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p258r\" (UniqueName: \"kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.752566 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.752517    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.752566 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.752541    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.853443 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.853383    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.853876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.853502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p258r\" (UniqueName: \"kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.853876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.853525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.853876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.853844    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.853876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.853851    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.863511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.863483    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p258r\" (UniqueName: \"kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:52.944175 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:52.944142    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:24:53.093528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.093503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"]
Jul 06 15:24:53.096500 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:53.096468    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7143d25_ef83_434a_a4f7_744aeeefcc6b.slice/crio-8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273 WatchSource:0}: Error finding container 8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273: Status 404 returned error can't find the container with id 8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273
Jul 06 15:24:53.228248 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.228208    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"]
Jul 06 15:24:53.232172 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.232149    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.241720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.241497    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"]
Jul 06 15:24:53.358606 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.358571    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.358802 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.358666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69psl\" (UniqueName: \"kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.358802 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.358745    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.431628 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.431593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n" event={"ID":"c7143d25-ef83-434a-a4f7-744aeeefcc6b","Type":"ContainerStarted","Data":"8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273"}
Jul 06 15:24:53.459911 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.459876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69psl\" (UniqueName: \"kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.460075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.459935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.460149 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.460115    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.460425 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.460379    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.460489 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.460430    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.470742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.470716    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69psl\" (UniqueName: \"kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.545292 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.545249    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:24:53.702207 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.702019    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"]
Jul 06 15:24:53.704462 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:53.704427    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53339932_c90e_4667_ad6b_02a8961d03d6.slice/crio-0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c WatchSource:0}: Error finding container 0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c: Status 404 returned error can't find the container with id 0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c
Jul 06 15:24:53.835280 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.835229    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"]
Jul 06 15:24:53.840371 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.840337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:53.849340 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.849304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"]
Jul 06 15:24:53.964772 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.964681    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:53.964772 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.964738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4r8\" (UniqueName: \"kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:53.964772 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:53.964774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.065815 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.065777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.065993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.065831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4r8\" (UniqueName: \"kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.065993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.065873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.066469 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.066433    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.066577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.066503    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.075580 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.075536    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4r8\" (UniqueName: \"kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.155444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.155383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:24:54.340381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.338822    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"]
Jul 06 15:24:54.344873 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:24:54.342918    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69537b75_910d_4214_bed3_1e61717f1b24.slice/crio-89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2 WatchSource:0}: Error finding container 89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2: Status 404 returned error can't find the container with id 89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2
Jul 06 15:24:54.439599 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.439536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m" event={"ID":"69537b75-910d-4214-bed3-1e61717f1b24","Type":"ContainerStarted","Data":"89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2"}
Jul 06 15:24:54.441053 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:24:54.441005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc" event={"ID":"53339932-c90e-4667-ad6b-02a8961d03d6","Type":"ContainerStarted","Data":"0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c"}
Jul 06 15:25:00.475516 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.475485    2566 generic.go:356] "Generic (PLEG): container finished" podID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerID="5e869952a215faa0580d2120cfee39c46e0f34f2e13e2e799460c7a02f6ae2f1" exitCode=0
Jul 06 15:25:00.475938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.475559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs" event={"ID":"9f50476d-c6c7-4766-bb5a-72a40ebaca96","Type":"ContainerDied","Data":"5e869952a215faa0580d2120cfee39c46e0f34f2e13e2e799460c7a02f6ae2f1"}
Jul 06 15:25:00.477032 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.477009    2566 generic.go:356] "Generic (PLEG): container finished" podID="69537b75-910d-4214-bed3-1e61717f1b24" containerID="235f707e6b2d98169a94c2992d5fa65441bc8191473d0623248a9ad4deaee351" exitCode=0
Jul 06 15:25:00.477145 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.477099    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m" event={"ID":"69537b75-910d-4214-bed3-1e61717f1b24","Type":"ContainerDied","Data":"235f707e6b2d98169a94c2992d5fa65441bc8191473d0623248a9ad4deaee351"}
Jul 06 15:25:00.478506 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.478475    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerID="feb9cc971e609e2312ef1ba1675005bed24a9025bb21c3c6a091058b46b13702" exitCode=0
Jul 06 15:25:00.478633 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.478501    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n" event={"ID":"c7143d25-ef83-434a-a4f7-744aeeefcc6b","Type":"ContainerDied","Data":"feb9cc971e609e2312ef1ba1675005bed24a9025bb21c3c6a091058b46b13702"}
Jul 06 15:25:00.480234 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.480215    2566 generic.go:356] "Generic (PLEG): container finished" podID="53339932-c90e-4667-ad6b-02a8961d03d6" containerID="058b24f5b62f952b3e7f0e0a37c9015c638acb4d96fcbc388fc63885f87b5696" exitCode=0
Jul 06 15:25:00.480327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:00.480250    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc" event={"ID":"53339932-c90e-4667-ad6b-02a8961d03d6","Type":"ContainerDied","Data":"058b24f5b62f952b3e7f0e0a37c9015c638acb4d96fcbc388fc63885f87b5696"}
Jul 06 15:25:02.491292 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.491209    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerID="3f390cc3facb3527a79457dc5710a59d6212bf8d02dcd1c3b6b2e8b0947d3684" exitCode=0
Jul 06 15:25:02.491737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.491297    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n" event={"ID":"c7143d25-ef83-434a-a4f7-744aeeefcc6b","Type":"ContainerDied","Data":"3f390cc3facb3527a79457dc5710a59d6212bf8d02dcd1c3b6b2e8b0947d3684"}
Jul 06 15:25:02.493258 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.493231    2566 generic.go:356] "Generic (PLEG): container finished" podID="53339932-c90e-4667-ad6b-02a8961d03d6" containerID="f82d2efba0142d8062c30a556c08936947e3279b7c78922a21a57ed66da23087" exitCode=0
Jul 06 15:25:02.493344 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.493320    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc" event={"ID":"53339932-c90e-4667-ad6b-02a8961d03d6","Type":"ContainerDied","Data":"f82d2efba0142d8062c30a556c08936947e3279b7c78922a21a57ed66da23087"}
Jul 06 15:25:02.495096 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.495075    2566 generic.go:356] "Generic (PLEG): container finished" podID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerID="918efc6532133af0922f2d5d2e6f37626819d0f2ee7f33dda2f512f8087deb6a" exitCode=0
Jul 06 15:25:02.495219 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.495149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs" event={"ID":"9f50476d-c6c7-4766-bb5a-72a40ebaca96","Type":"ContainerDied","Data":"918efc6532133af0922f2d5d2e6f37626819d0f2ee7f33dda2f512f8087deb6a"}
Jul 06 15:25:02.497259 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.497237    2566 generic.go:356] "Generic (PLEG): container finished" podID="69537b75-910d-4214-bed3-1e61717f1b24" containerID="043b8f833e0b1c632d6a002bfa2d79d794de62e94d329002a862e4040325a3c8" exitCode=0
Jul 06 15:25:02.497331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:02.497270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m" event={"ID":"69537b75-910d-4214-bed3-1e61717f1b24","Type":"ContainerDied","Data":"043b8f833e0b1c632d6a002bfa2d79d794de62e94d329002a862e4040325a3c8"}
Jul 06 15:25:11.550422 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.550366    2566 generic.go:356] "Generic (PLEG): container finished" podID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerID="69e851e11ba8a9587f9fdf9e4f5e78e74b8164cc11272bf4fc0de914e9fab499" exitCode=0
Jul 06 15:25:11.550882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.550456    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs" event={"ID":"9f50476d-c6c7-4766-bb5a-72a40ebaca96","Type":"ContainerDied","Data":"69e851e11ba8a9587f9fdf9e4f5e78e74b8164cc11272bf4fc0de914e9fab499"}
Jul 06 15:25:11.552303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.552279    2566 generic.go:356] "Generic (PLEG): container finished" podID="69537b75-910d-4214-bed3-1e61717f1b24" containerID="13e978799417450fc9f749941abf109fc61d397f8fe04874d3e2b96d5ff959ec" exitCode=0
Jul 06 15:25:11.552428 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.552328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m" event={"ID":"69537b75-910d-4214-bed3-1e61717f1b24","Type":"ContainerDied","Data":"13e978799417450fc9f749941abf109fc61d397f8fe04874d3e2b96d5ff959ec"}
Jul 06 15:25:11.554358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.554338    2566 generic.go:356] "Generic (PLEG): container finished" podID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerID="efbe3372aaedf11b762cac4aff7ed1725a17242029c1e4ab30d99664c26f6d95" exitCode=0
Jul 06 15:25:11.554501 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.554430    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n" event={"ID":"c7143d25-ef83-434a-a4f7-744aeeefcc6b","Type":"ContainerDied","Data":"efbe3372aaedf11b762cac4aff7ed1725a17242029c1e4ab30d99664c26f6d95"}
Jul 06 15:25:11.556284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.556264    2566 generic.go:356] "Generic (PLEG): container finished" podID="53339932-c90e-4667-ad6b-02a8961d03d6" containerID="1ae5e744c011fc31fd41f5cb73b61533b4afcee367a19b4252ac18e26f282a87" exitCode=0
Jul 06 15:25:11.556367 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:11.556303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc" event={"ID":"53339932-c90e-4667-ad6b-02a8961d03d6","Type":"ContainerDied","Data":"1ae5e744c011fc31fd41f5cb73b61533b4afcee367a19b4252ac18e26f282a87"}
Jul 06 15:25:12.706542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.706504    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:25:12.711081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.711041    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util\") pod \"69537b75-910d-4214-bed3-1e61717f1b24\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") "
Jul 06 15:25:12.711221 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.711181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kn4r8\" (UniqueName: \"kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8\") pod \"69537b75-910d-4214-bed3-1e61717f1b24\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") "
Jul 06 15:25:12.711281 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.711252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle\") pod \"69537b75-910d-4214-bed3-1e61717f1b24\" (UID: \"69537b75-910d-4214-bed3-1e61717f1b24\") "
Jul 06 15:25:12.711928 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.711886    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle" (OuterVolumeSpecName: "bundle") pod "69537b75-910d-4214-bed3-1e61717f1b24" (UID: "69537b75-910d-4214-bed3-1e61717f1b24"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.714835 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.714796    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8" (OuterVolumeSpecName: "kube-api-access-kn4r8") pod "69537b75-910d-4214-bed3-1e61717f1b24" (UID: "69537b75-910d-4214-bed3-1e61717f1b24"). InnerVolumeSpecName "kube-api-access-kn4r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:12.718422 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.718351    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util" (OuterVolumeSpecName: "util") pod "69537b75-910d-4214-bed3-1e61717f1b24" (UID: "69537b75-910d-4214-bed3-1e61717f1b24"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.744606 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.744575    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:25:12.761329 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.761305    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:25:12.770490 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.770455    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:25:12.812352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812276    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle\") pod \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") "
Jul 06 15:25:12.812352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812320    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util\") pod \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") "
Jul 06 15:25:12.812352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812352    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69psl\" (UniqueName: \"kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl\") pod \"53339932-c90e-4667-ad6b-02a8961d03d6\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812377    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util\") pod \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812435    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util\") pod \"53339932-c90e-4667-ad6b-02a8961d03d6\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdjvb\" (UniqueName: \"kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb\") pod \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\" (UID: \"9f50476d-c6c7-4766-bb5a-72a40ebaca96\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812523    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle\") pod \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812548    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle\") pod \"53339932-c90e-4667-ad6b-02a8961d03d6\" (UID: \"53339932-c90e-4667-ad6b-02a8961d03d6\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p258r\" (UniqueName: \"kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r\") pod \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\" (UID: \"c7143d25-ef83-434a-a4f7-744aeeefcc6b\") "
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812802    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4r8\" (UniqueName: \"kubernetes.io/projected/69537b75-910d-4214-bed3-1e61717f1b24-kube-api-access-kn4r8\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812817    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-bundle\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812830    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/69537b75-910d-4214-bed3-1e61717f1b24-util\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.813044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.812900    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle" (OuterVolumeSpecName: "bundle") pod "9f50476d-c6c7-4766-bb5a-72a40ebaca96" (UID: "9f50476d-c6c7-4766-bb5a-72a40ebaca96"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.813555 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.813338    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle" (OuterVolumeSpecName: "bundle") pod "c7143d25-ef83-434a-a4f7-744aeeefcc6b" (UID: "c7143d25-ef83-434a-a4f7-744aeeefcc6b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.813775 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.813752    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle" (OuterVolumeSpecName: "bundle") pod "53339932-c90e-4667-ad6b-02a8961d03d6" (UID: "53339932-c90e-4667-ad6b-02a8961d03d6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.815650 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.815623    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r" (OuterVolumeSpecName: "kube-api-access-p258r") pod "c7143d25-ef83-434a-a4f7-744aeeefcc6b" (UID: "c7143d25-ef83-434a-a4f7-744aeeefcc6b"). InnerVolumeSpecName "kube-api-access-p258r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:12.815940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.815909    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb" (OuterVolumeSpecName: "kube-api-access-zdjvb") pod "9f50476d-c6c7-4766-bb5a-72a40ebaca96" (UID: "9f50476d-c6c7-4766-bb5a-72a40ebaca96"). InnerVolumeSpecName "kube-api-access-zdjvb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:12.817514 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.817488    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl" (OuterVolumeSpecName: "kube-api-access-69psl") pod "53339932-c90e-4667-ad6b-02a8961d03d6" (UID: "53339932-c90e-4667-ad6b-02a8961d03d6"). InnerVolumeSpecName "kube-api-access-69psl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:12.817863 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.817827    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util" (OuterVolumeSpecName: "util") pod "9f50476d-c6c7-4766-bb5a-72a40ebaca96" (UID: "9f50476d-c6c7-4766-bb5a-72a40ebaca96"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.820890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.820864    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util" (OuterVolumeSpecName: "util") pod "c7143d25-ef83-434a-a4f7-744aeeefcc6b" (UID: "c7143d25-ef83-434a-a4f7-744aeeefcc6b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.820890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.820878    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util" (OuterVolumeSpecName: "util") pod "53339932-c90e-4667-ad6b-02a8961d03d6" (UID: "53339932-c90e-4667-ad6b-02a8961d03d6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:12.913528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913493    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-util\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913521    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69psl\" (UniqueName: \"kubernetes.io/projected/53339932-c90e-4667-ad6b-02a8961d03d6-kube-api-access-69psl\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913531    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-util\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913539    2566 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-util\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913546    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdjvb\" (UniqueName: \"kubernetes.io/projected/9f50476d-c6c7-4766-bb5a-72a40ebaca96-kube-api-access-zdjvb\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913555    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7143d25-ef83-434a-a4f7-744aeeefcc6b-bundle\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913564    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/53339932-c90e-4667-ad6b-02a8961d03d6-bundle\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913571    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p258r\" (UniqueName: \"kubernetes.io/projected/c7143d25-ef83-434a-a4f7-744aeeefcc6b-kube-api-access-p258r\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:12.913778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:12.913579    2566 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9f50476d-c6c7-4766-bb5a-72a40ebaca96-bundle\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:13.429823 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.429783    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cfdb5c74f-rw4lj"]
Jul 06 15:25:13.430046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430034    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="util"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430048    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="util"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430057    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="pull"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430062    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="pull"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430069    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="extract"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430076    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="extract"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430083    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="pull"
Jul 06 15:25:13.430089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430090    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="pull"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430097    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="pull"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430102    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="pull"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430113    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430118    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430125    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430130    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430137    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="pull"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430141    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="pull"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430152    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="util"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430159    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430164    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430169    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430173    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430183    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430187    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430227    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69537b75-910d-4214-bed3-1e61717f1b24" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430234    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="53339932-c90e-4667-ad6b-02a8961d03d6" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430241    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7143d25-ef83-434a-a4f7-744aeeefcc6b" containerName="extract"
Jul 06 15:25:13.430274 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.430246    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f50476d-c6c7-4766-bb5a-72a40ebaca96" containerName="extract"
Jul 06 15:25:13.461744 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.461712    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfdb5c74f-rw4lj"]
Jul 06 15:25:13.461876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.461807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.464951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.464922    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 06 15:25:13.464951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.464935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-7fhr8\""
Jul 06 15:25:13.466811 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.466787    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 06 15:25:13.467018 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.466999    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 06 15:25:13.467069 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.467062    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 06 15:25:13.467185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.467162    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 06 15:25:13.467890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.467870    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 06 15:25:13.468252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.468226    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 06 15:25:13.473350 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.473328    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 06 15:25:13.519525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519484    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-oauth-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-console-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-oauth-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-trusted-ca-bundle\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2g7d\" (UniqueName: \"kubernetes.io/projected/c967eb02-f4ec-46df-84cb-e2414569165b-kube-api-access-s2g7d\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.519742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.519732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-service-ca\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.567351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.567323    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n"
Jul 06 15:25:13.567351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.567337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n" event={"ID":"c7143d25-ef83-434a-a4f7-744aeeefcc6b","Type":"ContainerDied","Data":"8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273"}
Jul 06 15:25:13.567576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.567387    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edfe8bc80fd6eb6b580e0d31498423c51347be2964dbc0c3aa4e74c3d4e4273"
Jul 06 15:25:13.569276 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.569257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc"
Jul 06 15:25:13.569428 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.569280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc" event={"ID":"53339932-c90e-4667-ad6b-02a8961d03d6","Type":"ContainerDied","Data":"0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c"}
Jul 06 15:25:13.569428 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.569324    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c71c26bb4a620cb5fa978d9b4f9a6e820eb5888a9cf0f41986df492aa4b9e4c"
Jul 06 15:25:13.571360 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.571332    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs" event={"ID":"9f50476d-c6c7-4766-bb5a-72a40ebaca96","Type":"ContainerDied","Data":"03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861"}
Jul 06 15:25:13.571488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.571375    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f9c73f388a5cf4d999508b0fda7e8142cc8d7137f431f927f7435494926861"
Jul 06 15:25:13.571488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.571343    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs"
Jul 06 15:25:13.573231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.573208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m" event={"ID":"69537b75-910d-4214-bed3-1e61717f1b24","Type":"ContainerDied","Data":"89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2"}
Jul 06 15:25:13.573231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.573228    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m"
Jul 06 15:25:13.573417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.573246    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89ffe32e256689ae3018d622c4cd2b2a423486c0d9147cdacf83c20cd08a56b2"
Jul 06 15:25:13.620121 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620085    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620121 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-oauth-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-console-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-oauth-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-trusted-ca-bundle\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620222    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2g7d\" (UniqueName: \"kubernetes.io/projected/c967eb02-f4ec-46df-84cb-e2414569165b-kube-api-access-s2g7d\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.620352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-service-ca\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.621001 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.620971    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-oauth-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.621109 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.621062    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-console-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.621109 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.621098    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-service-ca\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.622433 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.621201    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c967eb02-f4ec-46df-84cb-e2414569165b-trusted-ca-bundle\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.622684 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.622658    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-oauth-config\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.623015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.622996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c967eb02-f4ec-46df-84cb-e2414569165b-console-serving-cert\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.630290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.630268    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2g7d\" (UniqueName: \"kubernetes.io/projected/c967eb02-f4ec-46df-84cb-e2414569165b-kube-api-access-s2g7d\") pod \"console-7cfdb5c74f-rw4lj\" (UID: \"c967eb02-f4ec-46df-84cb-e2414569165b\") " pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.771373 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.771294    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:13.956107 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:13.956072    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cfdb5c74f-rw4lj"]
Jul 06 15:25:13.959759 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:13.959733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc967eb02_f4ec_46df_84cb_e2414569165b.slice/crio-7b493ee08d05589ea6bab54330179ebc56c299bccaa44be2a8afb785fa2669fc WatchSource:0}: Error finding container 7b493ee08d05589ea6bab54330179ebc56c299bccaa44be2a8afb785fa2669fc: Status 404 returned error can't find the container with id 7b493ee08d05589ea6bab54330179ebc56c299bccaa44be2a8afb785fa2669fc
Jul 06 15:25:14.579454 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:14.579419    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfdb5c74f-rw4lj" event={"ID":"c967eb02-f4ec-46df-84cb-e2414569165b","Type":"ContainerStarted","Data":"7b493ee08d05589ea6bab54330179ebc56c299bccaa44be2a8afb785fa2669fc"}
Jul 06 15:25:17.596309 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:17.596275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cfdb5c74f-rw4lj" event={"ID":"c967eb02-f4ec-46df-84cb-e2414569165b","Type":"ContainerStarted","Data":"92949e083bf38ae82fd8f947eee01685753f8efaedea06d09c4eee8e9d06dc0c"}
Jul 06 15:25:17.618607 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:17.618558    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cfdb5c74f-rw4lj" podStartSLOduration=1.3362542290000001 podStartE2EDuration="4.61854133s" totalImagesPullingTime="3.282287101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:17.61712024 +0000 UTC m=+498.397549060" watchObservedRunningTime="2026-07-06 15:25:17.61854133 +0000 UTC m=+498.398970131"
Jul 06 15:25:21.381459 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.381422    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"]
Jul 06 15:25:21.383809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.383787    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.386517 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.386499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f5ggj\""
Jul 06 15:25:21.403592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.403567    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"]
Jul 06 15:25:21.480477 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.480438    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.480620 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.480499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9mzl\" (UniqueName: \"kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.581871 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.581831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9mzl\" (UniqueName: \"kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.582029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.581898    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.582255 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.582238    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.590824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.590789    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9mzl\" (UniqueName: \"kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.694245 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.694158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:21.844192 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:21.844158    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"]
Jul 06 15:25:21.846212 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:21.846186    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf47ecf5_eff0_477b_a55a_4b77cce90fa3.slice/crio-16b90bedf3872f945b51305f73c6c33d13df1e7143407430da7ace73b2c58182 WatchSource:0}: Error finding container 16b90bedf3872f945b51305f73c6c33d13df1e7143407430da7ace73b2c58182: Status 404 returned error can't find the container with id 16b90bedf3872f945b51305f73c6c33d13df1e7143407430da7ace73b2c58182
Jul 06 15:25:22.618252 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:22.618214    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c" event={"ID":"cf47ecf5-eff0-477b-a55a-4b77cce90fa3","Type":"ContainerStarted","Data":"16b90bedf3872f945b51305f73c6c33d13df1e7143407430da7ace73b2c58182"}
Jul 06 15:25:23.771882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.771841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:23.772279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.771906    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:23.776529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.776506    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:23.862087 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.862047    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"]
Jul 06 15:25:23.866655 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.866618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:23.873272 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.873242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-gtzhw\""
Jul 06 15:25:23.899761 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.899729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbqf\" (UniqueName: \"kubernetes.io/projected/5619760d-a735-4d20-9c8e-c83e5ea38129-kube-api-access-4nbqf\") pod \"authorino-operator-75cc8d5cb7-8mrzp\" (UID: \"5619760d-a735-4d20-9c8e-c83e5ea38129\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:23.905198 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:23.905162    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"]
Jul 06 15:25:24.000377 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.000340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbqf\" (UniqueName: \"kubernetes.io/projected/5619760d-a735-4d20-9c8e-c83e5ea38129-kube-api-access-4nbqf\") pod \"authorino-operator-75cc8d5cb7-8mrzp\" (UID: \"5619760d-a735-4d20-9c8e-c83e5ea38129\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:24.022556 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.022490    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbqf\" (UniqueName: \"kubernetes.io/projected/5619760d-a735-4d20-9c8e-c83e5ea38129-kube-api-access-4nbqf\") pod \"authorino-operator-75cc8d5cb7-8mrzp\" (UID: \"5619760d-a735-4d20-9c8e-c83e5ea38129\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:24.180155 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.180116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:24.320525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.320494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"]
Jul 06 15:25:24.322795 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:24.322772    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5619760d_a735_4d20_9c8e_c83e5ea38129.slice/crio-24502ad8bc6edcf966040b1c842c2a43cc57a994f404d763b077b2353670875b WatchSource:0}: Error finding container 24502ad8bc6edcf966040b1c842c2a43cc57a994f404d763b077b2353670875b: Status 404 returned error can't find the container with id 24502ad8bc6edcf966040b1c842c2a43cc57a994f404d763b077b2353670875b
Jul 06 15:25:24.629116 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.629079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp" event={"ID":"5619760d-a735-4d20-9c8e-c83e5ea38129","Type":"ContainerStarted","Data":"24502ad8bc6edcf966040b1c842c2a43cc57a994f404d763b077b2353670875b"}
Jul 06 15:25:24.633006 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:24.632981    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cfdb5c74f-rw4lj"
Jul 06 15:25:25.961918 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:25.961880    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"]
Jul 06 15:25:25.971414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:25.971370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:25.976235 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:25.975954    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-nwgr2\""
Jul 06 15:25:25.976235 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:25.976080    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 06 15:25:25.984407 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:25.984371    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"]
Jul 06 15:25:26.019890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:26.019858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z72j7\" (UniqueName: \"kubernetes.io/projected/99c9b368-3f11-4161-b7dc-5335411fe249-kube-api-access-z72j7\") pod \"dns-operator-controller-manager-864dd78c4c-lmlhw\" (UID: \"99c9b368-3f11-4161-b7dc-5335411fe249\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:26.120567 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:26.120526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z72j7\" (UniqueName: \"kubernetes.io/projected/99c9b368-3f11-4161-b7dc-5335411fe249-kube-api-access-z72j7\") pod \"dns-operator-controller-manager-864dd78c4c-lmlhw\" (UID: \"99c9b368-3f11-4161-b7dc-5335411fe249\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:26.131509 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:26.131477    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z72j7\" (UniqueName: \"kubernetes.io/projected/99c9b368-3f11-4161-b7dc-5335411fe249-kube-api-access-z72j7\") pod \"dns-operator-controller-manager-864dd78c4c-lmlhw\" (UID: \"99c9b368-3f11-4161-b7dc-5335411fe249\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:26.288799 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:26.288712    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:26.718327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:26.718189    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"]
Jul 06 15:25:26.721239 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:26.721213    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c9b368_3f11_4161_b7dc_5335411fe249.slice/crio-c78ca3d6dd3ed4fda0240efdae9cd6573f48a3f5de6f22685b54a2a5ea09efe3 WatchSource:0}: Error finding container c78ca3d6dd3ed4fda0240efdae9cd6573f48a3f5de6f22685b54a2a5ea09efe3: Status 404 returned error can't find the container with id c78ca3d6dd3ed4fda0240efdae9cd6573f48a3f5de6f22685b54a2a5ea09efe3
Jul 06 15:25:27.647093 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:27.647052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw" event={"ID":"99c9b368-3f11-4161-b7dc-5335411fe249","Type":"ContainerStarted","Data":"c78ca3d6dd3ed4fda0240efdae9cd6573f48a3f5de6f22685b54a2a5ea09efe3"}
Jul 06 15:25:27.648846 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:27.648818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp" event={"ID":"5619760d-a735-4d20-9c8e-c83e5ea38129","Type":"ContainerStarted","Data":"54d81b6b140b9160bb6e7fc885f2c6a8890cde5aa1a36d79d8f8b76116257931"}
Jul 06 15:25:27.648979 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:27.648959    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:27.673155 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:27.673098    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp" podStartSLOduration=2.380828161 podStartE2EDuration="4.673083209s" totalImagesPullingTime="2.292255048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:27.671684389 +0000 UTC m=+508.452113203" watchObservedRunningTime="2026-07-06 15:25:27.673083209 +0000 UTC m=+508.453512010"
Jul 06 15:25:28.532353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.532317    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"]
Jul 06 15:25:28.535944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.535920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:28.538854 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.538830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-bhhp6\""
Jul 06 15:25:28.553716 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.553684    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"]
Jul 06 15:25:28.641951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.641910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mdtm\" (UniqueName: \"kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm\") pod \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" (UID: \"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:28.743116 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.743081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mdtm\" (UniqueName: \"kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm\") pod \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" (UID: \"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:28.752717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.752688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mdtm\" (UniqueName: \"kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm\") pod \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" (UID: \"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:28.847925 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:28.847894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:29.206240 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:29.206208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"]
Jul 06 15:25:29.232689 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:29.232650    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723a1e1c_fe4c_45d7_ab69_10d4e184b04f.slice/crio-2cd1a56ff23b0c43a0a9b1b70a5a55dc4f7b805bd14d6ec4fab9ca23fecb8e0e WatchSource:0}: Error finding container 2cd1a56ff23b0c43a0a9b1b70a5a55dc4f7b805bd14d6ec4fab9ca23fecb8e0e: Status 404 returned error can't find the container with id 2cd1a56ff23b0c43a0a9b1b70a5a55dc4f7b805bd14d6ec4fab9ca23fecb8e0e
Jul 06 15:25:29.661425 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:29.661376    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c" event={"ID":"cf47ecf5-eff0-477b-a55a-4b77cce90fa3","Type":"ContainerStarted","Data":"47a3ad9eff5e1c07d33a80f31376a45e675ea8bddc3ae3756181155fb111e8c4"}
Jul 06 15:25:29.661605 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:29.661475    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:29.662813 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:29.662790    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq" event={"ID":"723a1e1c-fe4c-45d7-ab69-10d4e184b04f","Type":"ContainerStarted","Data":"2cd1a56ff23b0c43a0a9b1b70a5a55dc4f7b805bd14d6ec4fab9ca23fecb8e0e"}
Jul 06 15:25:29.683470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:29.683419    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c" podStartSLOduration=1.4757767560000001 podStartE2EDuration="8.683382049s" totalImagesPullingTime="7.207605293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:29.681683815 +0000 UTC m=+510.462112617" watchObservedRunningTime="2026-07-06 15:25:29.683382049 +0000 UTC m=+510.463810850"
Jul 06 15:25:30.670019 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:30.669921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw" event={"ID":"99c9b368-3f11-4161-b7dc-5335411fe249","Type":"ContainerStarted","Data":"949306ad11e097a98cf7e30b95088f4134c423138080f2d44180f9f8e27a7037"}
Jul 06 15:25:30.670596 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:30.670282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:30.691924 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:30.691866    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw" podStartSLOduration=2.541216376 podStartE2EDuration="5.691849029s" totalImagesPullingTime="3.150632653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:30.689710252 +0000 UTC m=+511.470139079" watchObservedRunningTime="2026-07-06 15:25:30.691849029 +0000 UTC m=+511.472277830"
Jul 06 15:25:31.675697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:31.675661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq" event={"ID":"723a1e1c-fe4c-45d7-ab69-10d4e184b04f","Type":"ContainerStarted","Data":"a435cfdc2d80e4a358664541fed2413573bc20c5638e32e3065039ce0ec911f4"}
Jul 06 15:25:31.676123 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:31.675874    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:31.696704 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:31.696652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq" podStartSLOduration=1.9174067259999998 podStartE2EDuration="3.696636766s" totalImagesPullingTime="1.77923004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:31.695183537 +0000 UTC m=+512.475612350" watchObservedRunningTime="2026-07-06 15:25:31.696636766 +0000 UTC m=+512.477065567"
Jul 06 15:25:32.716615 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.716578    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"]
Jul 06 15:25:32.720030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.720003    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.722645 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.722621    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 06 15:25:32.723444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.723422    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 06 15:25:32.723545 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.723478    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xr89z\""
Jul 06 15:25:32.729662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.729641    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"]
Jul 06 15:25:32.775773 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.775733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b974909-582b-4e4e-8bfc-dd80e0ed839f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.775969 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.775793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7l2k\" (UniqueName: \"kubernetes.io/projected/5b974909-582b-4e4e-8bfc-dd80e0ed839f-kube-api-access-j7l2k\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.775969 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.775920    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.876884 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.876851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.877046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.876900    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b974909-582b-4e4e-8bfc-dd80e0ed839f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.877046 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:32.876996    2566 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Jul 06 15:25:32.877046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.877012    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7l2k\" (UniqueName: \"kubernetes.io/projected/5b974909-582b-4e4e-8bfc-dd80e0ed839f-kube-api-access-j7l2k\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.877154 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:32.877064    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert podName:5b974909-582b-4e4e-8bfc-dd80e0ed839f nodeName:}" failed. No retries permitted until 2026-07-06 15:25:33.377044943 +0000 UTC m=+514.157473727 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-2nhqw" (UID: "5b974909-582b-4e4e-8bfc-dd80e0ed839f") : secret "plugin-serving-cert" not found
Jul 06 15:25:32.877488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.877473    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5b974909-582b-4e4e-8bfc-dd80e0ed839f-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:32.894451 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:32.894422    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7l2k\" (UniqueName: \"kubernetes.io/projected/5b974909-582b-4e4e-8bfc-dd80e0ed839f-kube-api-access-j7l2k\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:33.382573 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:33.382525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:33.384910 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:33.384880    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/5b974909-582b-4e4e-8bfc-dd80e0ed839f-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-2nhqw\" (UID: \"5b974909-582b-4e4e-8bfc-dd80e0ed839f\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:33.628982 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:33.628946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"
Jul 06 15:25:33.763169 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:33.763146    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw"]
Jul 06 15:25:33.765181 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:33.765154    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b974909_582b_4e4e_8bfc_dd80e0ed839f.slice/crio-0d926b0ba554192a9ff9eab54e59232f142da2986fb0be01258df3e700dc5e4a WatchSource:0}: Error finding container 0d926b0ba554192a9ff9eab54e59232f142da2986fb0be01258df3e700dc5e4a: Status 404 returned error can't find the container with id 0d926b0ba554192a9ff9eab54e59232f142da2986fb0be01258df3e700dc5e4a
Jul 06 15:25:34.692687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:34.692640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw" event={"ID":"5b974909-582b-4e4e-8bfc-dd80e0ed839f","Type":"ContainerStarted","Data":"0d926b0ba554192a9ff9eab54e59232f142da2986fb0be01258df3e700dc5e4a"}
Jul 06 15:25:38.657495 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:38.657451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:25:40.672227 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:40.672182    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:41.678715 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:41.678681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-lmlhw"
Jul 06 15:25:42.562093 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.562059    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"]
Jul 06 15:25:42.562302 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.562261    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" containerName="manager" containerID="cri-o://47a3ad9eff5e1c07d33a80f31376a45e675ea8bddc3ae3756181155fb111e8c4" gracePeriod=2
Jul 06 15:25:42.569094 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.569069    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"]
Jul 06 15:25:42.587212 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.587180    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"]
Jul 06 15:25:42.588012 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.587463    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" containerName="manager" containerID="cri-o://a435cfdc2d80e4a358664541fed2413573bc20c5638e32e3065039ce0ec911f4" gracePeriod=2
Jul 06 15:25:42.589674 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.589647    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:42.602089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.602059    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"]
Jul 06 15:25:42.607173 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.607140    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-9mgvq\": pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:42.609585 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.609562    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:25:42.609920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.609903    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" containerName="manager"
Jul 06 15:25:42.609998 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.609922    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" containerName="manager"
Jul 06 15:25:42.609998 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.609946    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" containerName="manager"
Jul 06 15:25:42.609998 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.609955    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" containerName="manager"
Jul 06 15:25:42.610207 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.610041    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" containerName="manager"
Jul 06 15:25:42.610207 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.610054    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" containerName="manager"
Jul 06 15:25:42.614035 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.614002    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:42.615953 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.615920    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:42.617941 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.617912    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:42.628813 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.628785    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.630354 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.630330    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:25:42.631091 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.631061    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:42.665502 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:42.665468    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:42.736125 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.736083    2566 generic.go:356] "Generic (PLEG): container finished" podID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" containerID="47a3ad9eff5e1c07d33a80f31376a45e675ea8bddc3ae3756181155fb111e8c4" exitCode=0
Jul 06 15:25:42.738042 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.738015    2566 generic.go:356] "Generic (PLEG): container finished" podID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" containerID="a435cfdc2d80e4a358664541fed2413573bc20c5638e32e3065039ce0ec911f4" exitCode=0
Jul 06 15:25:42.767532 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.767491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.767698 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.767587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzkcp\" (UniqueName: \"kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.868736 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.868694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzkcp\" (UniqueName: \"kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.868914 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.868783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.869260 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.869231    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:42.881314 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:42.881281    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzkcp\" (UniqueName: \"kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zr477\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:43.017158 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.017120    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:43.785701 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.785664    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:25:43.840372 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.840341    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:43.850980 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.850944    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:25:43.978087 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.978046    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:43.978087 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:43.978090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hznmk\" (UniqueName: \"kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:44.078583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:44.078501    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:44.078583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:44.078549    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hznmk\" (UniqueName: \"kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:44.078985 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:44.078966    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:44.088671 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:44.088644    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hznmk\" (UniqueName: \"kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk\") pod \"kuadrant-operator-controller-manager-86878c769f-92nd9\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:44.154505 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:44.154465    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:56.982241 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:56.982213    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:56.984864 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:56.984828    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:56.988026 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:56.987957    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:25:56.990066 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:56.990039    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod851bb383_f3d6_446a_818e_e1a189227284.slice/crio-d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3 WatchSource:0}: Error finding container d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3: Status 404 returned error can't find the container with id d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3
Jul 06 15:25:56.993287 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:56.993235    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:56.995665 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:56.995640    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:56.997832 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:56.997808    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:57.019341 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.019316    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:25:57.023438 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:25:57.023410    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b4c9d9_2f30_481d_9810_8c137a50a423.slice/crio-8664602d040e25cb4e59f816913d7f4c185b8f7ccd2eb6a378d9dd804669761b WatchSource:0}: Error finding container 8664602d040e25cb4e59f816913d7f4c185b8f7ccd2eb6a378d9dd804669761b: Status 404 returned error can't find the container with id 8664602d040e25cb4e59f816913d7f4c185b8f7ccd2eb6a378d9dd804669761b
Jul 06 15:25:57.095251 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.095208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume\") pod \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") "
Jul 06 15:25:57.095251 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.095250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mdtm\" (UniqueName: \"kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm\") pod \"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\" (UID: \"723a1e1c-fe4c-45d7-ab69-10d4e184b04f\") "
Jul 06 15:25:57.095506 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.095327    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9mzl\" (UniqueName: \"kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl\") pod \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\" (UID: \"cf47ecf5-eff0-477b-a55a-4b77cce90fa3\") "
Jul 06 15:25:57.095794 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.095764    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "cf47ecf5-eff0-477b-a55a-4b77cce90fa3" (UID: "cf47ecf5-eff0-477b-a55a-4b77cce90fa3"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:25:57.097822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.097796    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm" (OuterVolumeSpecName: "kube-api-access-8mdtm") pod "723a1e1c-fe4c-45d7-ab69-10d4e184b04f" (UID: "723a1e1c-fe4c-45d7-ab69-10d4e184b04f"). InnerVolumeSpecName "kube-api-access-8mdtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:57.097915 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.097827    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl" (OuterVolumeSpecName: "kube-api-access-n9mzl") pod "cf47ecf5-eff0-477b-a55a-4b77cce90fa3" (UID: "cf47ecf5-eff0-477b-a55a-4b77cce90fa3"). InnerVolumeSpecName "kube-api-access-n9mzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:25:57.195995 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.195957    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mdtm\" (UniqueName: \"kubernetes.io/projected/723a1e1c-fe4c-45d7-ab69-10d4e184b04f-kube-api-access-8mdtm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:57.195995 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.195999    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9mzl\" (UniqueName: \"kubernetes.io/projected/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-kube-api-access-n9mzl\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:57.196189 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.196016    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/cf47ecf5-eff0-477b-a55a-4b77cce90fa3-extensions-socket-volume\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:25:57.721908 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.721871    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" path="/var/lib/kubelet/pods/723a1e1c-fe4c-45d7-ab69-10d4e184b04f/volumes"
Jul 06 15:25:57.722214 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.722201    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" path="/var/lib/kubelet/pods/cf47ecf5-eff0-477b-a55a-4b77cce90fa3/volumes"
Jul 06 15:25:57.822896 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.822876    2566 scope.go:117] "RemoveContainer" containerID="a435cfdc2d80e4a358664541fed2413573bc20c5638e32e3065039ce0ec911f4"
Jul 06 15:25:57.823009 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.822893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:57.824875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.824824    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw" event={"ID":"5b974909-582b-4e4e-8bfc-dd80e0ed839f","Type":"ContainerStarted","Data":"2e7a923ce474f123541accd75e45936411434ea30f4d17a8eba61cbef2f95f9b"}
Jul 06 15:25:57.825714 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:57.825686    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:57.826677 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.826655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" event={"ID":"851bb383-f3d6-446a-818e-e1a189227284","Type":"ContainerStarted","Data":"f7b394000f501cc8bbf262f872be4aa6dac2277f0367f972f2ec6d8c729db922"}
Jul 06 15:25:57.826779 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.826705    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" event={"ID":"851bb383-f3d6-446a-818e-e1a189227284","Type":"ContainerStarted","Data":"d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3"}
Jul 06 15:25:57.826779 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.826738    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:25:57.827953 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:57.827927    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:57.828517 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.828498    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" event={"ID":"74b4c9d9-2f30-481d-9810-8c137a50a423","Type":"ContainerStarted","Data":"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"}
Jul 06 15:25:57.828606 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.828539    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" event={"ID":"74b4c9d9-2f30-481d-9810-8c137a50a423","Type":"ContainerStarted","Data":"8664602d040e25cb4e59f816913d7f4c185b8f7ccd2eb6a378d9dd804669761b"}
Jul 06 15:25:57.828686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.828661    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:25:57.829891 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.829873    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:57.835658 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.835629    2566 scope.go:117] "RemoveContainer" containerID="47a3ad9eff5e1c07d33a80f31376a45e675ea8bddc3ae3756181155fb111e8c4"
Jul 06 15:25:57.850087 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.850048    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-2nhqw" podStartSLOduration=2.697018339 podStartE2EDuration="25.850035926s" totalImagesPullingTime="23.153017587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:57.84831897 +0000 UTC m=+538.628747790" watchObservedRunningTime="2026-07-06 15:25:57.850035926 +0000 UTC m=+538.630465007"
Jul 06 15:25:57.875724 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.875673    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" podStartSLOduration=15.875658267 podStartE2EDuration="15.875658267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:57.86975743 +0000 UTC m=+538.650186249" watchObservedRunningTime="2026-07-06 15:25:57.875658267 +0000 UTC m=+538.656087069"
Jul 06 15:25:57.897058 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:57.897024    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:57.897592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:25:57.897539    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" podStartSLOduration=14.897520777 podStartE2EDuration="14.897520777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:25:57.895143509 +0000 UTC m=+538.675572309" watchObservedRunningTime="2026-07-06 15:25:57.897520777 +0000 UTC m=+538.677949583"
Jul 06 15:25:57.901015 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:57.900985    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:25:59.722179 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:59.722138    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-9mgvq\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="723a1e1c-fe4c-45d7-ab69-10d4e184b04f" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-9mgvq"
Jul 06 15:25:59.724629 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:25:59.724603    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l9z7c\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="cf47ecf5-eff0-477b-a55a-4b77cce90fa3" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l9z7c"
Jul 06 15:26:08.838138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:08.838101    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:26:08.838654 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:08.838410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:26:08.935639 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:08.935601    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:26:08.935855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:08.935806    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" podUID="74b4c9d9-2f30-481d-9810-8c137a50a423" containerName="manager" containerID="cri-o://4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c" gracePeriod=10
Jul 06 15:26:09.185092 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.185067    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:26:09.188735 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.188704    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzkcp\" (UniqueName: \"kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp\") pod \"74b4c9d9-2f30-481d-9810-8c137a50a423\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") "
Jul 06 15:26:09.188867 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.188755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume\") pod \"74b4c9d9-2f30-481d-9810-8c137a50a423\" (UID: \"74b4c9d9-2f30-481d-9810-8c137a50a423\") "
Jul 06 15:26:09.189138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.189116    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "74b4c9d9-2f30-481d-9810-8c137a50a423" (UID: "74b4c9d9-2f30-481d-9810-8c137a50a423"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:26:09.190829 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.190806    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp" (OuterVolumeSpecName: "kube-api-access-wzkcp") pod "74b4c9d9-2f30-481d-9810-8c137a50a423" (UID: "74b4c9d9-2f30-481d-9810-8c137a50a423"). InnerVolumeSpecName "kube-api-access-wzkcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:26:09.289279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.289245    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzkcp\" (UniqueName: \"kubernetes.io/projected/74b4c9d9-2f30-481d-9810-8c137a50a423-kube-api-access-wzkcp\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:26:09.289279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.289271    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74b4c9d9-2f30-481d-9810-8c137a50a423-extensions-socket-volume\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:26:09.890583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.890544    2566 generic.go:356] "Generic (PLEG): container finished" podID="74b4c9d9-2f30-481d-9810-8c137a50a423" containerID="4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c" exitCode=0
Jul 06 15:26:09.890583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.890585    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"
Jul 06 15:26:09.891185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.890628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" event={"ID":"74b4c9d9-2f30-481d-9810-8c137a50a423","Type":"ContainerDied","Data":"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"}
Jul 06 15:26:09.891185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.890673    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477" event={"ID":"74b4c9d9-2f30-481d-9810-8c137a50a423","Type":"ContainerDied","Data":"8664602d040e25cb4e59f816913d7f4c185b8f7ccd2eb6a378d9dd804669761b"}
Jul 06 15:26:09.891185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.890696    2566 scope.go:117] "RemoveContainer" containerID="4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"
Jul 06 15:26:09.901265 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.901248    2566 scope.go:117] "RemoveContainer" containerID="4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"
Jul 06 15:26:09.901544 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:26:09.901526    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c\": container with ID starting with 4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c not found: ID does not exist" containerID="4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"
Jul 06 15:26:09.901589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.901555    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c"} err="failed to get container status \"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c\": rpc error: code = NotFound desc = could not find container \"4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c\": container with ID starting with 4820ae593cc900b0a57d9f743e671d95abd5ef6f86e4f666e0619fa9a3f26c4c not found: ID does not exist"
Jul 06 15:26:09.927646 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.927613    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:26:09.936917 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:09.936892    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zr477"]
Jul 06 15:26:11.720524 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:11.720489    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74b4c9d9-2f30-481d-9810-8c137a50a423" path="/var/lib/kubelet/pods/74b4c9d9-2f30-481d-9810-8c137a50a423/volumes"
Jul 06 15:26:27.095209 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.095172    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:27.095614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.095480    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74b4c9d9-2f30-481d-9810-8c137a50a423" containerName="manager"
Jul 06 15:26:27.095614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.095493    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="74b4c9d9-2f30-481d-9810-8c137a50a423" containerName="manager"
Jul 06 15:26:27.095614 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.095553    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="74b4c9d9-2f30-481d-9810-8c137a50a423" containerName="manager"
Jul 06 15:26:27.162291 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.162258    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:27.162488 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.162374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.165253 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.165224    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 06 15:26:27.201411 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.201359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:27.230527 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.230495    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.230695 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.230552    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576sn\" (UniqueName: \"kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.331853 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.331815    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.332022 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.331877    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576sn\" (UniqueName: \"kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.332485 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.332467    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.341136 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.341110    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-576sn\" (UniqueName: \"kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn\") pod \"limitador-limitador-64c94fd64f-ff47f\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.475133 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.475050    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:27.565843 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.565806    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:26:27.569714 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.569691    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.595829 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.595794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:26:27.638174 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.638140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-config-file\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.638348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.638183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r98g7\" (UniqueName: \"kubernetes.io/projected/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-kube-api-access-r98g7\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.644045 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.644013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:26:27.649728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.649705    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:27.652714 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:26:27.652666    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9f38d1e_72b0_4a3a_a6b3_e48748610b48.slice/crio-ee4d90bbd4999a1e7328e161797166aaba5f849666b85f953339638e85ab2825 WatchSource:0}: Error finding container ee4d90bbd4999a1e7328e161797166aaba5f849666b85f953339638e85ab2825: Status 404 returned error can't find the container with id ee4d90bbd4999a1e7328e161797166aaba5f849666b85f953339638e85ab2825
Jul 06 15:26:27.738683 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.738593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-config-file\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.738683 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.738633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r98g7\" (UniqueName: \"kubernetes.io/projected/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-kube-api-access-r98g7\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.739263 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.739244    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-config-file\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.750456 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.750424    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r98g7\" (UniqueName: \"kubernetes.io/projected/bdc5bd1d-94c5-47a5-9e42-d776340b5c3d-kube-api-access-r98g7\") pod \"limitador-limitador-54d6d988d5-tl7gw\" (UID: \"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.880011 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.879974    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:27.972917 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:27.972878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" event={"ID":"d9f38d1e-72b0-4a3a-a6b3-e48748610b48","Type":"ContainerStarted","Data":"ee4d90bbd4999a1e7328e161797166aaba5f849666b85f953339638e85ab2825"}
Jul 06 15:26:28.018263 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.018235    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:26:28.020338 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:26:28.020311    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc5bd1d_94c5_47a5_9e42_d776340b5c3d.slice/crio-f2074167f154306f964fd9e44063017dec99664c235b9b22942ffe20f206e5f8 WatchSource:0}: Error finding container f2074167f154306f964fd9e44063017dec99664c235b9b22942ffe20f206e5f8: Status 404 returned error can't find the container with id f2074167f154306f964fd9e44063017dec99664c235b9b22942ffe20f206e5f8
Jul 06 15:26:28.042935 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.042907    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:28.047637 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.047612    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:28.050854 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.050679    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trg2f\""
Jul 06 15:26:28.054358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.054327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:28.142919 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.142880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkq7\" (UniqueName: \"kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7\") pod \"authorino-58cd955675-v858v\" (UID: \"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe\") " pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:28.244232 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.244193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkq7\" (UniqueName: \"kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7\") pod \"authorino-58cd955675-v858v\" (UID: \"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe\") " pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:28.261337 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.261308    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkq7\" (UniqueName: \"kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7\") pod \"authorino-58cd955675-v858v\" (UID: \"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe\") " pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:28.358266 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.358190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:28.493137 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.492905    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:28.495319 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:26:28.495297    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod035a23ea_1092_4b3f_8a1a_f80c2afe2cbe.slice/crio-7a16b1799559473d3c79899541a8137652c2af0ef1091211966aeba5a280a9c0 WatchSource:0}: Error finding container 7a16b1799559473d3c79899541a8137652c2af0ef1091211966aeba5a280a9c0: Status 404 returned error can't find the container with id 7a16b1799559473d3c79899541a8137652c2af0ef1091211966aeba5a280a9c0
Jul 06 15:26:28.985730 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.985690    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v858v" event={"ID":"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe","Type":"ContainerStarted","Data":"7a16b1799559473d3c79899541a8137652c2af0ef1091211966aeba5a280a9c0"}
Jul 06 15:26:28.988615 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:28.988580    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw" event={"ID":"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d","Type":"ContainerStarted","Data":"f2074167f154306f964fd9e44063017dec99664c235b9b22942ffe20f206e5f8"}
Jul 06 15:26:33.012464 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.012422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" event={"ID":"d9f38d1e-72b0-4a3a-a6b3-e48748610b48","Type":"ContainerStarted","Data":"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"}
Jul 06 15:26:33.013218 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.013194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:33.017506 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.017470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw" event={"ID":"bdc5bd1d-94c5-47a5-9e42-d776340b5c3d","Type":"ContainerStarted","Data":"680ae5cbcf47362e0b658a723e0b2b8960319b381fa16492af7cdf6bd460bc73"}
Jul 06 15:26:33.017660 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.017565    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:33.019367 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.019338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v858v" event={"ID":"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe","Type":"ContainerStarted","Data":"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"}
Jul 06 15:26:33.038608 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.038559    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" podStartSLOduration=0.844954983 podStartE2EDuration="6.038545662s" totalImagesPullingTime="5.193590679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:33.036915171 +0000 UTC m=+573.817343971" watchObservedRunningTime="2026-07-06 15:26:33.038545662 +0000 UTC m=+573.818974462"
Jul 06 15:26:33.066037 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.065984    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-v858v" podStartSLOduration=0.6536983 podStartE2EDuration="5.065969094s" totalImagesPullingTime="4.412270794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:33.065639393 +0000 UTC m=+573.846068213" watchObservedRunningTime="2026-07-06 15:26:33.065969094 +0000 UTC m=+573.846397894"
Jul 06 15:26:33.077449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.077417    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:33.090751 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:33.090689    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw" podStartSLOduration=1.26434707 podStartE2EDuration="6.090669774s" totalImagesPullingTime="4.826322704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:26:33.089286577 +0000 UTC m=+573.869715383" watchObservedRunningTime="2026-07-06 15:26:33.090669774 +0000 UTC m=+573.871098577"
Jul 06 15:26:35.028753 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:35.028716    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-v858v" podUID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" containerName="authorino" containerID="cri-o://1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57" gracePeriod=30
Jul 06 15:26:35.281880 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:35.281818    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:35.409813 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:35.409775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hkq7\" (UniqueName: \"kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7\") pod \"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe\" (UID: \"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe\") "
Jul 06 15:26:35.411883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:35.411842    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7" (OuterVolumeSpecName: "kube-api-access-4hkq7") pod "035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" (UID: "035a23ea-1092-4b3f-8a1a-f80c2afe2cbe"). InnerVolumeSpecName "kube-api-access-4hkq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:26:35.511238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:35.511200    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkq7\" (UniqueName: \"kubernetes.io/projected/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe-kube-api-access-4hkq7\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:26:36.034564 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.034480    2566 generic.go:356] "Generic (PLEG): container finished" podID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" containerID="1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57" exitCode=0
Jul 06 15:26:36.034564 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.034536    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-v858v"
Jul 06 15:26:36.034975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.034569    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v858v" event={"ID":"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe","Type":"ContainerDied","Data":"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"}
Jul 06 15:26:36.034975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.034612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-v858v" event={"ID":"035a23ea-1092-4b3f-8a1a-f80c2afe2cbe","Type":"ContainerDied","Data":"7a16b1799559473d3c79899541a8137652c2af0ef1091211966aeba5a280a9c0"}
Jul 06 15:26:36.034975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.034629    2566 scope.go:117] "RemoveContainer" containerID="1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"
Jul 06 15:26:36.042749 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.042724    2566 scope.go:117] "RemoveContainer" containerID="1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"
Jul 06 15:26:36.043024 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:26:36.043002    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57\": container with ID starting with 1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57 not found: ID does not exist" containerID="1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"
Jul 06 15:26:36.043116 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.043039    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57"} err="failed to get container status \"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57\": rpc error: code = NotFound desc = could not find container \"1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57\": container with ID starting with 1858e8520071a86967e62925a53ec975eaa6eb5dbdea770c0b4a70a1597e5d57 not found: ID does not exist"
Jul 06 15:26:36.054433 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.054404    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:36.062131 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:36.062108    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-v858v"]
Jul 06 15:26:37.721248 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:37.721203    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" path="/var/lib/kubelet/pods/035a23ea-1092-4b3f-8a1a-f80c2afe2cbe/volumes"
Jul 06 15:26:44.025969 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.025941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"
Jul 06 15:26:44.082175 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.082131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:44.082358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.082327    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" podUID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" containerName="limitador" containerID="cri-o://8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b" gracePeriod=30
Jul 06 15:26:44.083041 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.083018    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:44.636016 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.635992    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:44.789467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.789346    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file\") pod \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") "
Jul 06 15:26:44.789467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.789458    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-576sn\" (UniqueName: \"kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn\") pod \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\" (UID: \"d9f38d1e-72b0-4a3a-a6b3-e48748610b48\") "
Jul 06 15:26:44.789741 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.789710    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file" (OuterVolumeSpecName: "config-file") pod "d9f38d1e-72b0-4a3a-a6b3-e48748610b48" (UID: "d9f38d1e-72b0-4a3a-a6b3-e48748610b48"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:26:44.791446 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.791425    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn" (OuterVolumeSpecName: "kube-api-access-576sn") pod "d9f38d1e-72b0-4a3a-a6b3-e48748610b48" (UID: "d9f38d1e-72b0-4a3a-a6b3-e48748610b48"). InnerVolumeSpecName "kube-api-access-576sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:26:44.890360 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.890321    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-config-file\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:26:44.890360 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:44.890351    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576sn\" (UniqueName: \"kubernetes.io/projected/d9f38d1e-72b0-4a3a-a6b3-e48748610b48-kube-api-access-576sn\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:26:45.080282 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.080191    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" containerID="8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b" exitCode=0
Jul 06 15:26:45.080282 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.080237    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f"
Jul 06 15:26:45.080282 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.080256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" event={"ID":"d9f38d1e-72b0-4a3a-a6b3-e48748610b48","Type":"ContainerDied","Data":"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"}
Jul 06 15:26:45.080769 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.080304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-ff47f" event={"ID":"d9f38d1e-72b0-4a3a-a6b3-e48748610b48","Type":"ContainerDied","Data":"ee4d90bbd4999a1e7328e161797166aaba5f849666b85f953339638e85ab2825"}
Jul 06 15:26:45.080769 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.080335    2566 scope.go:117] "RemoveContainer" containerID="8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"
Jul 06 15:26:45.089031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.088893    2566 scope.go:117] "RemoveContainer" containerID="8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"
Jul 06 15:26:45.089168 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:26:45.089147    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b\": container with ID starting with 8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b not found: ID does not exist" containerID="8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"
Jul 06 15:26:45.089206 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.089177    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b"} err="failed to get container status \"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b\": rpc error: code = NotFound desc = could not find container \"8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b\": container with ID starting with 8c0f642ff6a96758fef04116e7bc9a9e9fd64ebed84deb2628a9a7fa46f6358b not found: ID does not exist"
Jul 06 15:26:45.102791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.102764    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:45.109646 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.109623    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-ff47f"]
Jul 06 15:26:45.720821 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:45.720789    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" path="/var/lib/kubelet/pods/d9f38d1e-72b0-4a3a-a6b3-e48748610b48/volumes"
Jul 06 15:26:48.694465 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694432    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-6255s"]
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694719    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" containerName="limitador"
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694729    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" containerName="limitador"
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694746    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" containerName="authorino"
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694752    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" containerName="authorino"
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694802    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="035a23ea-1092-4b3f-8a1a-f80c2afe2cbe" containerName="authorino"
Jul 06 15:26:48.694926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.694810    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9f38d1e-72b0-4a3a-a6b3-e48748610b48" containerName="limitador"
Jul 06 15:26:48.698890 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.698868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.701120 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.701100    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-877mr\""
Jul 06 15:26:48.702380 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.702356    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 15:26:48.710450 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.710425    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-6255s"]
Jul 06 15:26:48.822926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.822891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5f7bc1-b314-427d-9768-1d9bffe0fded-data\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.822926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.822937    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wjg\" (UniqueName: \"kubernetes.io/projected/bc5f7bc1-b314-427d-9768-1d9bffe0fded-kube-api-access-65wjg\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.924168 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.924126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wjg\" (UniqueName: \"kubernetes.io/projected/bc5f7bc1-b314-427d-9768-1d9bffe0fded-kube-api-access-65wjg\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.924418 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.924229    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5f7bc1-b314-427d-9768-1d9bffe0fded-data\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.924599 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.924582    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bc5f7bc1-b314-427d-9768-1d9bffe0fded-data\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:48.933068 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:48.933047    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wjg\" (UniqueName: \"kubernetes.io/projected/bc5f7bc1-b314-427d-9768-1d9bffe0fded-kube-api-access-65wjg\") pod \"postgres-69fb4594cf-6255s\" (UID: \"bc5f7bc1-b314-427d-9768-1d9bffe0fded\") " pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:49.008909 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:49.008825    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:26:49.138465 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:49.138387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-6255s"]
Jul 06 15:26:49.140813 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:26:49.140786    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5f7bc1_b314_427d_9768_1d9bffe0fded.slice/crio-f51c9e2188c4302459fa5cc1b620ebcdd22a80335da05d00fd9e5e0cbfb80709 WatchSource:0}: Error finding container f51c9e2188c4302459fa5cc1b620ebcdd22a80335da05d00fd9e5e0cbfb80709: Status 404 returned error can't find the container with id f51c9e2188c4302459fa5cc1b620ebcdd22a80335da05d00fd9e5e0cbfb80709
Jul 06 15:26:50.107365 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:26:50.107334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-6255s" event={"ID":"bc5f7bc1-b314-427d-9768-1d9bffe0fded","Type":"ContainerStarted","Data":"f51c9e2188c4302459fa5cc1b620ebcdd22a80335da05d00fd9e5e0cbfb80709"}
Jul 06 15:27:34.277829 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:34.277807    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\""
Jul 06 15:27:35.336239 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:35.336203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-6255s" event={"ID":"bc5f7bc1-b314-427d-9768-1d9bffe0fded","Type":"ContainerStarted","Data":"0231eb1903b9be5b4eb62be45505110e142fc979593b811f2160bbbbbdc45528"}
Jul 06 15:27:35.336682 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:35.336328    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:27:35.355583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:35.355530    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-6255s" podStartSLOduration=2.222838611 podStartE2EDuration="47.355516599s" totalImagesPullingTime="45.132677988s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:35.354274612 +0000 UTC m=+636.134703418" watchObservedRunningTime="2026-07-06 15:27:35.355516599 +0000 UTC m=+636.135945400"
Jul 06 15:27:41.370868 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:41.370837    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-6255s"
Jul 06 15:27:41.943681 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:41.943645    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:41.949164 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:41.949141    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:41.951578 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:41.951555    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-trg2f\""
Jul 06 15:27:41.953417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:41.953381    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:42.012926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.012890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d74c7\" (UniqueName: \"kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7\") pod \"authorino-5d4b67c446-4fnmp\" (UID: \"1678845a-325b-4a3c-92cd-f7caf6d1c00b\") " pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:42.113705 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.113655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d74c7\" (UniqueName: \"kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7\") pod \"authorino-5d4b67c446-4fnmp\" (UID: \"1678845a-325b-4a3c-92cd-f7caf6d1c00b\") " pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:42.129638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.129609    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d74c7\" (UniqueName: \"kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7\") pod \"authorino-5d4b67c446-4fnmp\" (UID: \"1678845a-325b-4a3c-92cd-f7caf6d1c00b\") " pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:42.191889 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.191858    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:42.192093 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.192082    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:42.232902 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.232871    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:42.239375 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.239352    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:42.251921 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.251387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:42.316065 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.316034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8n9g\" (UniqueName: \"kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g\") pod \"authorino-66769cfb79-r4z6h\" (UID: \"330d579c-488f-4e42-819a-cb2efa91404e\") " pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:42.338603 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.338578    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:42.340885 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:27:42.340860    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1678845a_325b_4a3c_92cd_f7caf6d1c00b.slice/crio-d9402885b7bc034efb57f3b1dcb783a083261372818220c1d8124fb9b206e5f8 WatchSource:0}: Error finding container d9402885b7bc034efb57f3b1dcb783a083261372818220c1d8124fb9b206e5f8: Status 404 returned error can't find the container with id d9402885b7bc034efb57f3b1dcb783a083261372818220c1d8124fb9b206e5f8
Jul 06 15:27:42.370824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.370783    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" event={"ID":"1678845a-325b-4a3c-92cd-f7caf6d1c00b","Type":"ContainerStarted","Data":"d9402885b7bc034efb57f3b1dcb783a083261372818220c1d8124fb9b206e5f8"}
Jul 06 15:27:42.417322 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.417282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8n9g\" (UniqueName: \"kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g\") pod \"authorino-66769cfb79-r4z6h\" (UID: \"330d579c-488f-4e42-819a-cb2efa91404e\") " pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:42.427104 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.427073    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8n9g\" (UniqueName: \"kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g\") pod \"authorino-66769cfb79-r4z6h\" (UID: \"330d579c-488f-4e42-819a-cb2efa91404e\") " pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:42.504892 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.504806    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:42.505105 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.505088    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:42.541529 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.541493    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:27:42.560830 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.560802    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:27:42.560980 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.560896    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.563995 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.563970    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 06 15:27:42.618728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.618691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.618898 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.618757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8wmk\" (UniqueName: \"kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.683810 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.683782    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:42.686073 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:27:42.686047    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod330d579c_488f_4e42_819a_cb2efa91404e.slice/crio-3083deba928a37bfb3a7663dba82c518e65dc74693c68b21539b7582d74d1f89 WatchSource:0}: Error finding container 3083deba928a37bfb3a7663dba82c518e65dc74693c68b21539b7582d74d1f89: Status 404 returned error can't find the container with id 3083deba928a37bfb3a7663dba82c518e65dc74693c68b21539b7582d74d1f89
Jul 06 15:27:42.719571 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.719546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8wmk\" (UniqueName: \"kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.719710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.719588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.721956 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.721933    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.729278 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.729260    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8wmk\" (UniqueName: \"kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk\") pod \"authorino-7b9f4f5c4c-pqh78\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") " pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:42.873235 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:42.873207    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:27:43.143368 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.143333    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:27:43.146576 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:27:43.146008    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod248243f8_8e6c_4944_a204_e4dbb96c1f5f.slice/crio-077f8426afa7ad5457c2135b5c48e46d0993a5d22a7ef954fbd1bba1037f1593 WatchSource:0}: Error finding container 077f8426afa7ad5457c2135b5c48e46d0993a5d22a7ef954fbd1bba1037f1593: Status 404 returned error can't find the container with id 077f8426afa7ad5457c2135b5c48e46d0993a5d22a7ef954fbd1bba1037f1593
Jul 06 15:27:43.377524 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.377486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" event={"ID":"1678845a-325b-4a3c-92cd-f7caf6d1c00b","Type":"ContainerStarted","Data":"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"}
Jul 06 15:27:43.377693 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.377535    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" podUID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" containerName="authorino" containerID="cri-o://fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3" gracePeriod=30
Jul 06 15:27:43.378713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.378688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66769cfb79-r4z6h" event={"ID":"330d579c-488f-4e42-819a-cb2efa91404e","Type":"ContainerStarted","Data":"3083deba928a37bfb3a7663dba82c518e65dc74693c68b21539b7582d74d1f89"}
Jul 06 15:27:43.379971 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.379952    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" event={"ID":"248243f8-8e6c-4944-a204-e4dbb96c1f5f","Type":"ContainerStarted","Data":"077f8426afa7ad5457c2135b5c48e46d0993a5d22a7ef954fbd1bba1037f1593"}
Jul 06 15:27:43.394506 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.394428    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" podStartSLOduration=1.725628878 podStartE2EDuration="2.39441249s" totalImagesPullingTime="668.783612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:43.393684386 +0000 UTC m=+644.174113205" watchObservedRunningTime="2026-07-06 15:27:43.39441249 +0000 UTC m=+644.174841282"
Jul 06 15:27:43.634710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.634685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:43.727106 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.727077    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d74c7\" (UniqueName: \"kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7\") pod \"1678845a-325b-4a3c-92cd-f7caf6d1c00b\" (UID: \"1678845a-325b-4a3c-92cd-f7caf6d1c00b\") "
Jul 06 15:27:43.729033 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.729011    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7" (OuterVolumeSpecName: "kube-api-access-d74c7") pod "1678845a-325b-4a3c-92cd-f7caf6d1c00b" (UID: "1678845a-325b-4a3c-92cd-f7caf6d1c00b"). InnerVolumeSpecName "kube-api-access-d74c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:27:43.828634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:43.828595    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d74c7\" (UniqueName: \"kubernetes.io/projected/1678845a-325b-4a3c-92cd-f7caf6d1c00b-kube-api-access-d74c7\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:27:44.386075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.385980    2566 generic.go:356] "Generic (PLEG): container finished" podID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" containerID="fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3" exitCode=0
Jul 06 15:27:44.386075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.386042    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-4fnmp"
Jul 06 15:27:44.386075 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.386062    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" event={"ID":"1678845a-325b-4a3c-92cd-f7caf6d1c00b","Type":"ContainerDied","Data":"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"}
Jul 06 15:27:44.386312 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.386102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-4fnmp" event={"ID":"1678845a-325b-4a3c-92cd-f7caf6d1c00b","Type":"ContainerDied","Data":"d9402885b7bc034efb57f3b1dcb783a083261372818220c1d8124fb9b206e5f8"}
Jul 06 15:27:44.386312 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.386118    2566 scope.go:117] "RemoveContainer" containerID="fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"
Jul 06 15:27:44.387593 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.387566    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66769cfb79-r4z6h" event={"ID":"330d579c-488f-4e42-819a-cb2efa91404e","Type":"ContainerStarted","Data":"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"}
Jul 06 15:27:44.387701 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.387622    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-66769cfb79-r4z6h" podUID="330d579c-488f-4e42-819a-cb2efa91404e" containerName="authorino" containerID="cri-o://d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a" gracePeriod=30
Jul 06 15:27:44.389303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.389278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" event={"ID":"248243f8-8e6c-4944-a204-e4dbb96c1f5f","Type":"ContainerStarted","Data":"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"}
Jul 06 15:27:44.395908 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.395890    2566 scope.go:117] "RemoveContainer" containerID="fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"
Jul 06 15:27:44.396204 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:27:44.396184    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3\": container with ID starting with fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3 not found: ID does not exist" containerID="fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"
Jul 06 15:27:44.396275 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.396210    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3"} err="failed to get container status \"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3\": rpc error: code = NotFound desc = could not find container \"fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3\": container with ID starting with fd201ce4cd7b2071366d4f9946892c09dad08239fdf76b1205c8990732179cb3 not found: ID does not exist"
Jul 06 15:27:44.409044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.408996    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-66769cfb79-r4z6h" podStartSLOduration=1.8110694889999999 podStartE2EDuration="2.408980617s" totalImagesPullingTime="597.911128ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:44.407544575 +0000 UTC m=+645.187973395" watchObservedRunningTime="2026-07-06 15:27:44.408980617 +0000 UTC m=+645.189409419"
Jul 06 15:27:44.428101 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.428052    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" podStartSLOduration=1.952456936 podStartE2EDuration="2.428036959s" totalImagesPullingTime="475.580023ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:27:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:27:44.427118929 +0000 UTC m=+645.207547748" watchObservedRunningTime="2026-07-06 15:27:44.428036959 +0000 UTC m=+645.208465759"
Jul 06 15:27:44.446435 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.446376    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:44.452539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.452509    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-4fnmp"]
Jul 06 15:27:44.631583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.631555    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:44.735792 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.735713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8n9g\" (UniqueName: \"kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g\") pod \"330d579c-488f-4e42-819a-cb2efa91404e\" (UID: \"330d579c-488f-4e42-819a-cb2efa91404e\") "
Jul 06 15:27:44.737753 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.737726    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g" (OuterVolumeSpecName: "kube-api-access-d8n9g") pod "330d579c-488f-4e42-819a-cb2efa91404e" (UID: "330d579c-488f-4e42-819a-cb2efa91404e"). InnerVolumeSpecName "kube-api-access-d8n9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:27:44.836537 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:44.836505    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8n9g\" (UniqueName: \"kubernetes.io/projected/330d579c-488f-4e42-819a-cb2efa91404e-kube-api-access-d8n9g\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:27:45.395753 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.395720    2566 generic.go:356] "Generic (PLEG): container finished" podID="330d579c-488f-4e42-819a-cb2efa91404e" containerID="d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a" exitCode=0
Jul 06 15:27:45.395955 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.395761    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-66769cfb79-r4z6h"
Jul 06 15:27:45.395955 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.395805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66769cfb79-r4z6h" event={"ID":"330d579c-488f-4e42-819a-cb2efa91404e","Type":"ContainerDied","Data":"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"}
Jul 06 15:27:45.395955 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.395849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-66769cfb79-r4z6h" event={"ID":"330d579c-488f-4e42-819a-cb2efa91404e","Type":"ContainerDied","Data":"3083deba928a37bfb3a7663dba82c518e65dc74693c68b21539b7582d74d1f89"}
Jul 06 15:27:45.395955 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.395872    2566 scope.go:117] "RemoveContainer" containerID="d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"
Jul 06 15:27:45.405040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.405024    2566 scope.go:117] "RemoveContainer" containerID="d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"
Jul 06 15:27:45.405283 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:27:45.405267    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a\": container with ID starting with d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a not found: ID does not exist" containerID="d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"
Jul 06 15:27:45.405317 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.405293    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a"} err="failed to get container status \"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a\": rpc error: code = NotFound desc = could not find container \"d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a\": container with ID starting with d26ddd3bec57abf5cea0a2bb592d5f0742f5ea50af66187a88531c672867924a not found: ID does not exist"
Jul 06 15:27:45.419421 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.419381    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:45.423777 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.423751    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-66769cfb79-r4z6h"]
Jul 06 15:27:45.721854 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.721774    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" path="/var/lib/kubelet/pods/1678845a-325b-4a3c-92cd-f7caf6d1c00b/volumes"
Jul 06 15:27:45.722085 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:27:45.722073    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="330d579c-488f-4e42-819a-cb2efa91404e" path="/var/lib/kubelet/pods/330d579c-488f-4e42-819a-cb2efa91404e/volumes"
Jul 06 15:28:27.167109 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.167077    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:28:27.213618 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213582    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-k92n7"]
Jul 06 15:28:27.213894 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" containerName="authorino"
Jul 06 15:28:27.213938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213896    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" containerName="authorino"
Jul 06 15:28:27.213938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213912    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="330d579c-488f-4e42-819a-cb2efa91404e" containerName="authorino"
Jul 06 15:28:27.213938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213918    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="330d579c-488f-4e42-819a-cb2efa91404e" containerName="authorino"
Jul 06 15:28:27.214027 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213976    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="330d579c-488f-4e42-819a-cb2efa91404e" containerName="authorino"
Jul 06 15:28:27.214027 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.213985    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1678845a-325b-4a3c-92cd-f7caf6d1c00b" containerName="authorino"
Jul 06 15:28:27.217077 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.217060    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.220288 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.220261    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 06 15:28:27.220444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.220419    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-swqh6\""
Jul 06 15:28:27.227101 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.227047    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-k92n7"]
Jul 06 15:28:27.378967 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.378935    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ed339-df5d-4a23-baed-989780818609-plugins-config-volume\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.379133 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.378979    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfgxn\" (UniqueName: \"kubernetes.io/projected/d14ed339-df5d-4a23-baed-989780818609-kube-api-access-vfgxn\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.480067 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.479990    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfgxn\" (UniqueName: \"kubernetes.io/projected/d14ed339-df5d-4a23-baed-989780818609-kube-api-access-vfgxn\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.480224 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.480068    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ed339-df5d-4a23-baed-989780818609-plugins-config-volume\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.480759 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.480740    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d14ed339-df5d-4a23-baed-989780818609-plugins-config-volume\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.491191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.491171    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfgxn\" (UniqueName: \"kubernetes.io/projected/d14ed339-df5d-4a23-baed-989780818609-kube-api-access-vfgxn\") pod \"payload-processing-58f86c7798-k92n7\" (UID: \"d14ed339-df5d-4a23-baed-989780818609\") " pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.528158 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.528128    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:27.697530 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.697507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-k92n7"]
Jul 06 15:28:27.699936 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:28:27.699910    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd14ed339_df5d_4a23_baed_989780818609.slice/crio-9cc2e3a7f69a85169e6fae3ee951c519244af38582b0bd12fa413b17ceef43bf WatchSource:0}: Error finding container 9cc2e3a7f69a85169e6fae3ee951c519244af38582b0bd12fa413b17ceef43bf: Status 404 returned error can't find the container with id 9cc2e3a7f69a85169e6fae3ee951c519244af38582b0bd12fa413b17ceef43bf
Jul 06 15:28:27.702281 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:27.702261    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:28:28.602456 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:28.602422    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-k92n7" event={"ID":"d14ed339-df5d-4a23-baed-989780818609","Type":"ContainerStarted","Data":"9cc2e3a7f69a85169e6fae3ee951c519244af38582b0bd12fa413b17ceef43bf"}
Jul 06 15:28:29.609231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:29.609202    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-k92n7" event={"ID":"d14ed339-df5d-4a23-baed-989780818609","Type":"ContainerStarted","Data":"57064127cc6199dd8ec6cbcbfe4666db0261156bb0bf1956992d8185e4341a7b"}
Jul 06 15:28:29.609641 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:29.609334    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:29.637486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:29.637436    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-k92n7" podStartSLOduration=1.262346178 podStartE2EDuration="2.637416648s" totalImagesPullingTime="1.37507047s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:28:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:28:29.6356272 +0000 UTC m=+690.416056003" watchObservedRunningTime="2026-07-06 15:28:29.637416648 +0000 UTC m=+690.417845455"
Jul 06 15:28:40.617020 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:40.616987    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-k92n7"
Jul 06 15:28:52.664108 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:28:52.664030    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:29:01.857383 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.857349    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"]
Jul 06 15:29:01.861682 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.861650    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.864523 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.864490    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 06 15:29:01.864640 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.864554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8k4vd\""
Jul 06 15:29:01.865351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.865329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 06 15:29:01.865519 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.865472    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 06 15:29:01.880086 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.880059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"]
Jul 06 15:29:01.951196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.951196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951199    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433c233-fd1f-4cb3-97ea-813a2964d259-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.951486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951227    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.951486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.951486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951330    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:01.951486 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:01.951462    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpmnn\" (UniqueName: \"kubernetes.io/projected/8433c233-fd1f-4cb3-97ea-813a2964d259-kube-api-access-qpmnn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.052727 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpmnn\" (UniqueName: \"kubernetes.io/projected/8433c233-fd1f-4cb3-97ea-813a2964d259-kube-api-access-qpmnn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.052940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.052940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052857    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433c233-fd1f-4cb3-97ea-813a2964d259-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.052940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.052940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052935    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.053140 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.052960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.053180 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.053171    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.053307 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.053284    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.053343 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.053325    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.055187 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.055168    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8433c233-fd1f-4cb3-97ea-813a2964d259-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.055342 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.055324    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8433c233-fd1f-4cb3-97ea-813a2964d259-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.061746 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.061710    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpmnn\" (UniqueName: \"kubernetes.io/projected/8433c233-fd1f-4cb3-97ea-813a2964d259-kube-api-access-qpmnn\") pod \"e2e-distinct-simulated-kserve-59975d69f9-g4c44\" (UID: \"8433c233-fd1f-4cb3-97ea-813a2964d259\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.172196 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.172107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:02.309949 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.309917    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"]
Jul 06 15:29:02.312640 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:29:02.312613    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8433c233_fd1f_4cb3_97ea_813a2964d259.slice/crio-6c5ba7794fdb7a9b6d88e822a0affd1f3b7a45b7c1cb711af13201b73bf7fa24 WatchSource:0}: Error finding container 6c5ba7794fdb7a9b6d88e822a0affd1f3b7a45b7c1cb711af13201b73bf7fa24: Status 404 returned error can't find the container with id 6c5ba7794fdb7a9b6d88e822a0affd1f3b7a45b7c1cb711af13201b73bf7fa24
Jul 06 15:29:02.663163 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.663122    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:29:02.773764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:02.773682    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44" event={"ID":"8433c233-fd1f-4cb3-97ea-813a2964d259","Type":"ContainerStarted","Data":"6c5ba7794fdb7a9b6d88e822a0affd1f3b7a45b7c1cb711af13201b73bf7fa24"}
Jul 06 15:29:03.785813 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:03.785717    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44" event={"ID":"8433c233-fd1f-4cb3-97ea-813a2964d259","Type":"ContainerStarted","Data":"9755b6e58d20a0023d194da6b348761e34b1f77e6f05fd97351a00b74a4b267c"}
Jul 06 15:29:03.808833 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:03.808775    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44" podStartSLOduration=1.625265616 podStartE2EDuration="2.808754682s" totalImagesPullingTime="1.183489066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:03.808718137 +0000 UTC m=+724.589146963" watchObservedRunningTime="2026-07-06 15:29:03.808754682 +0000 UTC m=+724.589183484"
Jul 06 15:29:12.172464 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:12.172424    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:12.172945 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:12.172486    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:12.185357 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:12.185322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:12.844168 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:12.844135    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-g4c44"
Jul 06 15:29:13.446436 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:13.446383    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:29:23.038888 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:23.038851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:29:38.439750 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:38.439707    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:29:52.438287 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.438253    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"]
Jul 06 15:29:52.441910 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.441888    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.444507 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.444479    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 06 15:29:52.456387 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.456359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"]
Jul 06 15:29:52.580029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.579989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.580029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.580030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.580297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.580054    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.580297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.580159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhr5v\" (UniqueName: \"kubernetes.io/projected/f8075fa4-6faf-4026-8cb4-39d2b108df52-kube-api-access-mhr5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.580297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.580214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8075fa4-6faf-4026-8cb4-39d2b108df52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.580297 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.580249    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681158 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhr5v\" (UniqueName: \"kubernetes.io/projected/f8075fa4-6faf-4026-8cb4-39d2b108df52-kube-api-access-mhr5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681174    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8075fa4-6faf-4026-8cb4-39d2b108df52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681304    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681727 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681698    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681812 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681739    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.681864 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.681810    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.683745 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.683721    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f8075fa4-6faf-4026-8cb4-39d2b108df52-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.684000 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.683977    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f8075fa4-6faf-4026-8cb4-39d2b108df52-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.691247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.691167    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhr5v\" (UniqueName: \"kubernetes.io/projected/f8075fa4-6faf-4026-8cb4-39d2b108df52-kube-api-access-mhr5v\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5\" (UID: \"f8075fa4-6faf-4026-8cb4-39d2b108df52\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.750747 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.750710    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:29:52.886445 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:52.886414    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"]
Jul 06 15:29:52.889241 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:29:52.889213    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8075fa4_6faf_4026_8cb4_39d2b108df52.slice/crio-c682b80c45657e77148af2f6c1618236351618424cd191ce8ed315fed3b48afb WatchSource:0}: Error finding container c682b80c45657e77148af2f6c1618236351618424cd191ce8ed315fed3b48afb: Status 404 returned error can't find the container with id c682b80c45657e77148af2f6c1618236351618424cd191ce8ed315fed3b48afb
Jul 06 15:29:53.028321 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:53.028235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5" event={"ID":"f8075fa4-6faf-4026-8cb4-39d2b108df52","Type":"ContainerStarted","Data":"c682b80c45657e77148af2f6c1618236351618424cd191ce8ed315fed3b48afb"}
Jul 06 15:29:54.034657 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:54.034622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5" event={"ID":"f8075fa4-6faf-4026-8cb4-39d2b108df52","Type":"ContainerStarted","Data":"1143d49d774f17d68a138f65fe8d9ce308118ece8a9f9339df2f07c379a588e6"}
Jul 06 15:29:54.052737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:54.052690    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5" podStartSLOduration=1.862079692 podStartE2EDuration="2.052675233s" totalImagesPullingTime="190.595541ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:29:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:29:54.052040716 +0000 UTC m=+774.832469544" watchObservedRunningTime="2026-07-06 15:29:54.052675233 +0000 UTC m=+774.833104065"
Jul 06 15:29:54.845997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:29:54.845963    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:30:00.138071 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.138037    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722530-hcs7s"]
Jul 06 15:30:00.140643 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.140619    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:00.143366 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.143345    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\""
Jul 06 15:30:00.145733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.145706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9cl\" (UniqueName: \"kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl\") pod \"maas-api-key-cleanup-29722530-hcs7s\" (UID: \"1969d50c-103b-41f4-90c8-7a5176beff2d\") " pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:00.150003 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.149980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722530-hcs7s"]
Jul 06 15:30:00.247015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.246979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9cl\" (UniqueName: \"kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl\") pod \"maas-api-key-cleanup-29722530-hcs7s\" (UID: \"1969d50c-103b-41f4-90c8-7a5176beff2d\") " pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:00.256931 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.256908    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9cl\" (UniqueName: \"kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl\") pod \"maas-api-key-cleanup-29722530-hcs7s\" (UID: \"1969d50c-103b-41f4-90c8-7a5176beff2d\") " pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:00.451678 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.451603    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:00.787522 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:00.787424    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722530-hcs7s"]
Jul 06 15:30:00.790221 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:30:00.790195    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1969d50c_103b_41f4_90c8_7a5176beff2d.slice/crio-4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f WatchSource:0}: Error finding container 4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f: Status 404 returned error can't find the container with id 4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f
Jul 06 15:30:01.070597 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:01.070478    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s" event={"ID":"1969d50c-103b-41f4-90c8-7a5176beff2d","Type":"ContainerStarted","Data":"4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f"}
Jul 06 15:30:02.751101 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:02.751061    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:30:02.751578 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:02.751114    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:30:02.765484 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:02.765451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:30:03.096314 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:03.096285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5"
Jul 06 15:30:04.089962 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:04.089904    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s" event={"ID":"1969d50c-103b-41f4-90c8-7a5176beff2d","Type":"ContainerStarted","Data":"5df266fabbb577ca1bb5ba5fd5e088582445352b7ec1631e092e72862833ecb3"}
Jul 06 15:30:04.104489 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:04.104436    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s" podStartSLOduration=1.721838012 podStartE2EDuration="4.104418266s" totalImagesPullingTime="2.382580254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:30:04.103860973 +0000 UTC m=+784.884289764" watchObservedRunningTime="2026-07-06 15:30:04.104418266 +0000 UTC m=+784.884847058"
Jul 06 15:30:05.096235 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:05.096199    2566 generic.go:356] "Generic (PLEG): container finished" podID="1969d50c-103b-41f4-90c8-7a5176beff2d" containerID="5df266fabbb577ca1bb5ba5fd5e088582445352b7ec1631e092e72862833ecb3" exitCode=0
Jul 06 15:30:05.096638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:05.096282    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s" event={"ID":"1969d50c-103b-41f4-90c8-7a5176beff2d","Type":"ContainerDied","Data":"5df266fabbb577ca1bb5ba5fd5e088582445352b7ec1631e092e72862833ecb3"}
Jul 06 15:30:06.230105 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:06.230079    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:30:06.290419 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:06.290364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm9cl\" (UniqueName: \"kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl\") pod \"1969d50c-103b-41f4-90c8-7a5176beff2d\" (UID: \"1969d50c-103b-41f4-90c8-7a5176beff2d\") "
Jul 06 15:30:06.292603 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:06.292572    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl" (OuterVolumeSpecName: "kube-api-access-lm9cl") pod "1969d50c-103b-41f4-90c8-7a5176beff2d" (UID: "1969d50c-103b-41f4-90c8-7a5176beff2d"). InnerVolumeSpecName "kube-api-access-lm9cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:30:06.391212 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:06.391122    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9cl\" (UniqueName: \"kubernetes.io/projected/1969d50c-103b-41f4-90c8-7a5176beff2d-kube-api-access-lm9cl\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:30:07.107953 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:07.107916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s" event={"ID":"1969d50c-103b-41f4-90c8-7a5176beff2d","Type":"ContainerDied","Data":"4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f"}
Jul 06 15:30:07.108126 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:07.107967    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dc47ac8241241526cd3ffd78e0839e5843fd565c2087af5610b2f6e9738459f"
Jul 06 15:30:07.108126 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:30:07.107928    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722530-hcs7s"
Jul 06 15:31:29.436649 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.436608    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7f854fbf54-wqcm6"]
Jul 06 15:31:29.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.437114    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1969d50c-103b-41f4-90c8-7a5176beff2d" containerName="cleanup"
Jul 06 15:31:29.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.437136    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1969d50c-103b-41f4-90c8-7a5176beff2d" containerName="cleanup"
Jul 06 15:31:29.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.437222    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1969d50c-103b-41f4-90c8-7a5176beff2d" containerName="cleanup"
Jul 06 15:31:29.439958 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.439937    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.442505 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:31:29.442472    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"authorino-7f854fbf54-wqcm6\" is forbidden: User \"system:node:ip-10-0-135-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-135-27.ec2.internal' and this object" podUID="05816ef8-1db3-4de0-a4e2-5c6724f1ae65" pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.461110 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.461082    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f854fbf54-wqcm6"]
Jul 06 15:31:29.488592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.488549    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txt92\" (UniqueName: \"kubernetes.io/projected/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-kube-api-access-txt92\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.488762 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.488615    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-tls-cert\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.589745 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.589707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txt92\" (UniqueName: \"kubernetes.io/projected/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-kube-api-access-txt92\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.589912 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.589753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-tls-cert\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.592053 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.592030    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-tls-cert\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.602634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.602608    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txt92\" (UniqueName: \"kubernetes.io/projected/05816ef8-1db3-4de0-a4e2-5c6724f1ae65-kube-api-access-txt92\") pod \"authorino-7f854fbf54-wqcm6\" (UID: \"05816ef8-1db3-4de0-a4e2-5c6724f1ae65\") " pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.748200 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.748121    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7f854fbf54-wqcm6"
Jul 06 15:31:29.900647 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:29.900617    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7f854fbf54-wqcm6"]
Jul 06 15:31:29.901997 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:31:29.901962    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05816ef8_1db3_4de0_a4e2_5c6724f1ae65.slice/crio-242e9b20fbf493750747a4aaa0a9a787ee30133ae689084e9f32b50cd424009c WatchSource:0}: Error finding container 242e9b20fbf493750747a4aaa0a9a787ee30133ae689084e9f32b50cd424009c: Status 404 returned error can't find the container with id 242e9b20fbf493750747a4aaa0a9a787ee30133ae689084e9f32b50cd424009c
Jul 06 15:31:30.507188 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.507150    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f854fbf54-wqcm6" event={"ID":"05816ef8-1db3-4de0-a4e2-5c6724f1ae65","Type":"ContainerStarted","Data":"b1946bc7b00036d50d85e2a593966a9dc2958492948d38507b0c85c583021e01"}
Jul 06 15:31:30.507689 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.507203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7f854fbf54-wqcm6" event={"ID":"05816ef8-1db3-4de0-a4e2-5c6724f1ae65","Type":"ContainerStarted","Data":"242e9b20fbf493750747a4aaa0a9a787ee30133ae689084e9f32b50cd424009c"}
Jul 06 15:31:30.531371 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.531321    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7f854fbf54-wqcm6" podStartSLOduration=1.111630492 podStartE2EDuration="1.531305501s" totalImagesPullingTime="419.675009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:31:30.529489384 +0000 UTC m=+871.309918188" watchObservedRunningTime="2026-07-06 15:31:30.531305501 +0000 UTC m=+871.311734303"
Jul 06 15:31:30.573227 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.573129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:31:30.573369 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.573345    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" podUID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" containerName="authorino" containerID="cri-o://7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a" gracePeriod=30
Jul 06 15:31:30.823273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.823194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:31:30.901768 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.901734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert\") pod \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") "
Jul 06 15:31:30.901934 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.901802    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8wmk\" (UniqueName: \"kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk\") pod \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\" (UID: \"248243f8-8e6c-4944-a204-e4dbb96c1f5f\") "
Jul 06 15:31:30.903920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.903884    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk" (OuterVolumeSpecName: "kube-api-access-q8wmk") pod "248243f8-8e6c-4944-a204-e4dbb96c1f5f" (UID: "248243f8-8e6c-4944-a204-e4dbb96c1f5f"). InnerVolumeSpecName "kube-api-access-q8wmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:31:30.911800 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:30.911779    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "248243f8-8e6c-4944-a204-e4dbb96c1f5f" (UID: "248243f8-8e6c-4944-a204-e4dbb96c1f5f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:31:31.003070 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.003031    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/248243f8-8e6c-4944-a204-e4dbb96c1f5f-tls-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:31:31.003070 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.003065    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8wmk\" (UniqueName: \"kubernetes.io/projected/248243f8-8e6c-4944-a204-e4dbb96c1f5f-kube-api-access-q8wmk\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:31:31.513774 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.513734    2566 generic.go:356] "Generic (PLEG): container finished" podID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" containerID="7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a" exitCode=0
Jul 06 15:31:31.513774 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.513776    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78"
Jul 06 15:31:31.514284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.513777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" event={"ID":"248243f8-8e6c-4944-a204-e4dbb96c1f5f","Type":"ContainerDied","Data":"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"}
Jul 06 15:31:31.514284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.513819    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7b9f4f5c4c-pqh78" event={"ID":"248243f8-8e6c-4944-a204-e4dbb96c1f5f","Type":"ContainerDied","Data":"077f8426afa7ad5457c2135b5c48e46d0993a5d22a7ef954fbd1bba1037f1593"}
Jul 06 15:31:31.514284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.513837    2566 scope.go:117] "RemoveContainer" containerID="7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"
Jul 06 15:31:31.523905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.523883    2566 scope.go:117] "RemoveContainer" containerID="7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"
Jul 06 15:31:31.524190 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:31:31.524169    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a\": container with ID starting with 7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a not found: ID does not exist" containerID="7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"
Jul 06 15:31:31.524260 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.524199    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a"} err="failed to get container status \"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a\": rpc error: code = NotFound desc = could not find container \"7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a\": container with ID starting with 7f617d521dd906536c7c6b4ef0cc520d41def1dec89e2e2d41bf0b83d10c0a3a not found: ID does not exist"
Jul 06 15:31:31.541687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.541660    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:31:31.558122 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.558094    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7b9f4f5c4c-pqh78"]
Jul 06 15:31:31.722829 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:31.722795    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" path="/var/lib/kubelet/pods/248243f8-8e6c-4944-a204-e4dbb96c1f5f/volumes"
Jul 06 15:31:59.660687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:31:59.660593    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:32:08.763975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:32:08.763938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:32:21.463594 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:32:21.463557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:32:29.073467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:32:29.073432    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:32:39.877848 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:32:39.877813    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:33:07.734352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:33:07.734320    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:33:20.581249 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:33:20.581208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:00.216783 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:00.216751    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:09.662977 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:09.662938    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:18.291506 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:18.291473    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:27.081059 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:27.081013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:39.203379 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:39.203342    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:34:49.379136 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:34:49.379100    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:35:00.449595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:35:00.449562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:35:09.889336 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:35:09.889296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:35:31.981716 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:35:31.981675    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:35:44.164007 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:35:44.163972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:36:10.414493 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:36:10.414451    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:36:32.742117 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:36:32.742042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:36:43.269327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:36:43.269291    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:36:54.146224 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:36:54.146183    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:37:03.258352 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:03.258313    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:37:12.353142 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:12.353110    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:37:22.249021 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:22.248609    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:37:48.554385 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:48.554349    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:37:48.556704 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:48.554601    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" podUID="851bb383-f3d6-446a-818e-e1a189227284" containerName="manager" containerID="cri-o://f7b394000f501cc8bbf262f872be4aa6dac2277f0367f972f2ec6d8c729db922" gracePeriod=10
Jul 06 15:37:51.397687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.397657    2566 generic.go:356] "Generic (PLEG): container finished" podID="851bb383-f3d6-446a-818e-e1a189227284" containerID="f7b394000f501cc8bbf262f872be4aa6dac2277f0367f972f2ec6d8c729db922" exitCode=0
Jul 06 15:37:51.397993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.397726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" event={"ID":"851bb383-f3d6-446a-818e-e1a189227284","Type":"ContainerDied","Data":"f7b394000f501cc8bbf262f872be4aa6dac2277f0367f972f2ec6d8c729db922"}
Jul 06 15:37:51.397993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.397776    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9" event={"ID":"851bb383-f3d6-446a-818e-e1a189227284","Type":"ContainerDied","Data":"d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3"}
Jul 06 15:37:51.397993 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.397792    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d70e0b6bda68217e340573b6f9289374dc52b9e3ffab20eeb0dc30cfa6fe47c3"
Jul 06 15:37:51.404314 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.404296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:37:51.504064 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.503989    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume\") pod \"851bb383-f3d6-446a-818e-e1a189227284\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") "
Jul 06 15:37:51.504064 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.504047    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hznmk\" (UniqueName: \"kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk\") pod \"851bb383-f3d6-446a-818e-e1a189227284\" (UID: \"851bb383-f3d6-446a-818e-e1a189227284\") "
Jul 06 15:37:51.504409 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.504367    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "851bb383-f3d6-446a-818e-e1a189227284" (UID: "851bb383-f3d6-446a-818e-e1a189227284"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:37:51.505996 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.505971    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk" (OuterVolumeSpecName: "kube-api-access-hznmk") pod "851bb383-f3d6-446a-818e-e1a189227284" (UID: "851bb383-f3d6-446a-818e-e1a189227284"). InnerVolumeSpecName "kube-api-access-hznmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:37:51.605553 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.605522    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/851bb383-f3d6-446a-818e-e1a189227284-extensions-socket-volume\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:37:51.605553 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:51.605549    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hznmk\" (UniqueName: \"kubernetes.io/projected/851bb383-f3d6-446a-818e-e1a189227284-kube-api-access-hznmk\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:37:52.402638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:52.402568    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"
Jul 06 15:37:52.422205 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:52.422175    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:37:52.427868 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:52.427844    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-92nd9"]
Jul 06 15:37:53.721262 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:37:53.721225    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="851bb383-f3d6-446a-818e-e1a189227284" path="/var/lib/kubelet/pods/851bb383-f3d6-446a-818e-e1a189227284/volumes"
Jul 06 15:38:00.126733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:00.126702    2566 scope.go:117] "RemoveContainer" containerID="f7b394000f501cc8bbf262f872be4aa6dac2277f0367f972f2ec6d8c729db922"
Jul 06 15:38:49.734791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.734757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"]
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735048    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" containerName="authorino"
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735058    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" containerName="authorino"
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735074    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="851bb383-f3d6-446a-818e-e1a189227284" containerName="manager"
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735079    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="851bb383-f3d6-446a-818e-e1a189227284" containerName="manager"
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735140    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="248243f8-8e6c-4944-a204-e4dbb96c1f5f" containerName="authorino"
Jul 06 15:38:49.735247 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.735147    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="851bb383-f3d6-446a-818e-e1a189227284" containerName="manager"
Jul 06 15:38:49.737965 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.737948    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.740957 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.740941    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-f5ggj\""
Jul 06 15:38:49.756812 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.756787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"]
Jul 06 15:38:49.798483 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.798449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc46494d-2858-4471-b045-6bc9a40efc20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.798647 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.798561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4296w\" (UniqueName: \"kubernetes.io/projected/dc46494d-2858-4471-b045-6bc9a40efc20-kube-api-access-4296w\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.899788 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.899735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc46494d-2858-4471-b045-6bc9a40efc20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.899788 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.899799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4296w\" (UniqueName: \"kubernetes.io/projected/dc46494d-2858-4471-b045-6bc9a40efc20-kube-api-access-4296w\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.900127 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.900105    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc46494d-2858-4471-b045-6bc9a40efc20-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:49.910207 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:49.910187    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4296w\" (UniqueName: \"kubernetes.io/projected/dc46494d-2858-4471-b045-6bc9a40efc20-kube-api-access-4296w\") pod \"kuadrant-operator-controller-manager-86878c769f-cdl6n\" (UID: \"dc46494d-2858-4471-b045-6bc9a40efc20\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:50.047458 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.047350    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:50.187556 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.187525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"]
Jul 06 15:38:50.190974 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:38:50.190944    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc46494d_2858_4471_b045_6bc9a40efc20.slice/crio-424aa71e615a54a4b51af8529b210cb7dd492efe30858d560b59ceed5a21052d WatchSource:0}: Error finding container 424aa71e615a54a4b51af8529b210cb7dd492efe30858d560b59ceed5a21052d: Status 404 returned error can't find the container with id 424aa71e615a54a4b51af8529b210cb7dd492efe30858d560b59ceed5a21052d
Jul 06 15:38:50.678633 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.678600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n" event={"ID":"dc46494d-2858-4471-b045-6bc9a40efc20","Type":"ContainerStarted","Data":"ef1fad360af1e4c0190db08bfb734cda67b226f736688c326f2b690264bd5c71"}
Jul 06 15:38:50.678818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.678650    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n" event={"ID":"dc46494d-2858-4471-b045-6bc9a40efc20","Type":"ContainerStarted","Data":"424aa71e615a54a4b51af8529b210cb7dd492efe30858d560b59ceed5a21052d"}
Jul 06 15:38:50.678818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.678695    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:38:50.713509 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:38:50.713459    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n" podStartSLOduration=1.713440523 podStartE2EDuration="1.713440523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:38:50.713019271 +0000 UTC m=+1311.493448071" watchObservedRunningTime="2026-07-06 15:38:50.713440523 +0000 UTC m=+1311.493869324"
Jul 06 15:39:01.685825 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:01.685745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-cdl6n"
Jul 06 15:39:10.361596 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:10.361562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:39:19.457444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:19.457404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:39:31.560593 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:31.560504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:39:43.473938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:43.473903    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:39:52.759847 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:39:52.759809    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:05.977099 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:05.977062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:14.155320 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:14.155284    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:23.956977 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:23.956939    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:35.558668 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:35.558633    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:44.650089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:44.650053    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:40:55.687333 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:40:55.687242    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:41:16.966981 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:41:16.966935    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:41:26.489270 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:41:26.489227    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:41:36.268884 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:41:36.268839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:41:47.855432 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:41:47.855379    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:42:05.502500 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:42:05.502464    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:42:19.763578 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:42:19.763545    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:42:28.889709 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:42:28.889666    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:42:42.522081 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:42:42.522042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:42:51.656755 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:42:51.656718    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:43:06.167833 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:43:06.167785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:43:15.350689 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:43:15.350651    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:43:24.862157 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:43:24.862123    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:43:36.765313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:43:36.765274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:43:54.676757 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:43:54.676680    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:44:07.468322 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:44:07.468290    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:44:19.154787 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:44:19.154742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:44:28.757546 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:44:28.757511    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:45:00.170103 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.170053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-2h48w"]
Jul 06 15:45:00.173964 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.173941    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:00.174090 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.174074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:00.177355 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.177336    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:00.177754 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.177740    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\""
Jul 06 15:45:00.194259 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.194232    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-2h48w"]
Jul 06 15:45:00.204408 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.204363    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:00.267381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.267344    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl4lj\" (UniqueName: \"kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj\") pod \"maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr\" (UID: \"dd7d8276-97fb-449c-91e1-1a014868a46b\") " pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:00.267580 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.267436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhtz\" (UniqueName: \"kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz\") pod \"maas-api-key-cleanup-29722545-2h48w\" (UID: \"070be589-012f-4f23-b573-c8c03fbf0628\") " pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:00.368885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.368842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl4lj\" (UniqueName: \"kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj\") pod \"maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr\" (UID: \"dd7d8276-97fb-449c-91e1-1a014868a46b\") " pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:00.369113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.368944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhtz\" (UniqueName: \"kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz\") pod \"maas-api-key-cleanup-29722545-2h48w\" (UID: \"070be589-012f-4f23-b573-c8c03fbf0628\") " pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:00.392027 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.391996    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhtz\" (UniqueName: \"kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz\") pod \"maas-api-key-cleanup-29722545-2h48w\" (UID: \"070be589-012f-4f23-b573-c8c03fbf0628\") " pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:00.402714 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.402688    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl4lj\" (UniqueName: \"kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj\") pod \"maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr\" (UID: \"dd7d8276-97fb-449c-91e1-1a014868a46b\") " pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:00.487381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.487278    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:00.494654 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.494631    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:00.690144 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.690115    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722545-2h48w"]
Jul 06 15:45:00.692385 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:45:00.692356    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod070be589_012f_4f23_b573_c8c03fbf0628.slice/crio-cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4 WatchSource:0}: Error finding container cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4: Status 404 returned error can't find the container with id cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4
Jul 06 15:45:00.882284 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:00.882255    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:00.883986 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:45:00.883958    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd7d8276_97fb_449c_91e1_1a014868a46b.slice/crio-cd9a770c3490f3c89fd7130e816ac69f96e4d5cffe188abf4708108d426865ae WatchSource:0}: Error finding container cd9a770c3490f3c89fd7130e816ac69f96e4d5cffe188abf4708108d426865ae: Status 404 returned error can't find the container with id cd9a770c3490f3c89fd7130e816ac69f96e4d5cffe188abf4708108d426865ae
Jul 06 15:45:01.407078 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.407039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:01.498226 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.498139    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w" event={"ID":"070be589-012f-4f23-b573-c8c03fbf0628","Type":"ContainerStarted","Data":"2c6f1b8fbe019b484830d6b4524a7b0d33652c7c0539610d612efafe3199d4f6"}
Jul 06 15:45:01.498226 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.498209    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w" event={"ID":"070be589-012f-4f23-b573-c8c03fbf0628","Type":"ContainerStarted","Data":"cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4"}
Jul 06 15:45:01.499833 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.499808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr" event={"ID":"dd7d8276-97fb-449c-91e1-1a014868a46b","Type":"ContainerStarted","Data":"c23fafc62b6fe7ee11d48106a4d85456b74833bfe4a877bbda0d0050949bbc45"}
Jul 06 15:45:01.499936 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.499849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr" event={"ID":"dd7d8276-97fb-449c-91e1-1a014868a46b","Type":"ContainerStarted","Data":"cd9a770c3490f3c89fd7130e816ac69f96e4d5cffe188abf4708108d426865ae"}
Jul 06 15:45:01.551033 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:01.550974    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w" podStartSLOduration=1.550957982 podStartE2EDuration="1.550957982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:45:01.55058888 +0000 UTC m=+1682.331017681" watchObservedRunningTime="2026-07-06 15:45:01.550957982 +0000 UTC m=+1682.331386782"
Jul 06 15:45:02.505755 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.505720    2566 generic.go:356] "Generic (PLEG): container finished" podID="dd7d8276-97fb-449c-91e1-1a014868a46b" containerID="c23fafc62b6fe7ee11d48106a4d85456b74833bfe4a877bbda0d0050949bbc45" exitCode=6
Jul 06 15:45:02.506183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.505810    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr" event={"ID":"dd7d8276-97fb-449c-91e1-1a014868a46b","Type":"ContainerDied","Data":"c23fafc62b6fe7ee11d48106a4d85456b74833bfe4a877bbda0d0050949bbc45"}
Jul 06 15:45:02.507429 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.507387    2566 generic.go:356] "Generic (PLEG): container finished" podID="070be589-012f-4f23-b573-c8c03fbf0628" containerID="2c6f1b8fbe019b484830d6b4524a7b0d33652c7c0539610d612efafe3199d4f6" exitCode=0
Jul 06 15:45:02.507528 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.507431    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w" event={"ID":"070be589-012f-4f23-b573-c8c03fbf0628","Type":"ContainerDied","Data":"2c6f1b8fbe019b484830d6b4524a7b0d33652c7c0539610d612efafe3199d4f6"}
Jul 06 15:45:02.639803 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.639778    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:02.791911 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.791820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl4lj\" (UniqueName: \"kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj\") pod \"dd7d8276-97fb-449c-91e1-1a014868a46b\" (UID: \"dd7d8276-97fb-449c-91e1-1a014868a46b\") "
Jul 06 15:45:02.793911 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.793885    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj" (OuterVolumeSpecName: "kube-api-access-dl4lj") pod "dd7d8276-97fb-449c-91e1-1a014868a46b" (UID: "dd7d8276-97fb-449c-91e1-1a014868a46b"). InnerVolumeSpecName "kube-api-access-dl4lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:45:02.893071 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:02.893029    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl4lj\" (UniqueName: \"kubernetes.io/projected/dd7d8276-97fb-449c-91e1-1a014868a46b-kube-api-access-dl4lj\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:45:03.513185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.513157    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"
Jul 06 15:45:03.513590 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.513194    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr" event={"ID":"dd7d8276-97fb-449c-91e1-1a014868a46b","Type":"ContainerDied","Data":"cd9a770c3490f3c89fd7130e816ac69f96e4d5cffe188abf4708108d426865ae"}
Jul 06 15:45:03.513692 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.513676    2566 scope.go:117] "RemoveContainer" containerID="c23fafc62b6fe7ee11d48106a4d85456b74833bfe4a877bbda0d0050949bbc45"
Jul 06 15:45:03.554417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.554344    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:03.556630 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.556604    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-ait-438fd2f6-29722545-zfrsr"]
Jul 06 15:45:03.660119 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.660086    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:03.722307 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.722262    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd7d8276-97fb-449c-91e1-1a014868a46b" path="/var/lib/kubelet/pods/dd7d8276-97fb-449c-91e1-1a014868a46b/volumes"
Jul 06 15:45:03.800856 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.800770    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9zhtz\" (UniqueName: \"kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz\") pod \"070be589-012f-4f23-b573-c8c03fbf0628\" (UID: \"070be589-012f-4f23-b573-c8c03fbf0628\") "
Jul 06 15:45:03.802818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.802783    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz" (OuterVolumeSpecName: "kube-api-access-9zhtz") pod "070be589-012f-4f23-b573-c8c03fbf0628" (UID: "070be589-012f-4f23-b573-c8c03fbf0628"). InnerVolumeSpecName "kube-api-access-9zhtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:45:03.901403 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:03.901362    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zhtz\" (UniqueName: \"kubernetes.io/projected/070be589-012f-4f23-b573-c8c03fbf0628-kube-api-access-9zhtz\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:45:04.526571 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:04.526540    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w"
Jul 06 15:45:04.526997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:04.526574    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722545-2h48w" event={"ID":"070be589-012f-4f23-b573-c8c03fbf0628","Type":"ContainerDied","Data":"cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4"}
Jul 06 15:45:04.526997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:45:04.526622    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc881729100bd64145188352cfbdd97b7c2fd37fd5948799602840a2a52874a4"
Jul 06 15:46:54.857957 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:46:54.857882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:47:03.656408 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:47:03.656364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:47:17.457465 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:47:17.457388    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:47:27.857627 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:47:27.857591    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:48:00.662154 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:00.662119    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:48:18.951844 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.951807    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:48:18.952223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952160    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070be589-012f-4f23-b573-c8c03fbf0628" containerName="cleanup"
Jul 06 15:48:18.952223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952171    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="070be589-012f-4f23-b573-c8c03fbf0628" containerName="cleanup"
Jul 06 15:48:18.952223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952186    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd7d8276-97fb-449c-91e1-1a014868a46b" containerName="cleanup"
Jul 06 15:48:18.952223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952191    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd7d8276-97fb-449c-91e1-1a014868a46b" containerName="cleanup"
Jul 06 15:48:18.952358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952252    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="070be589-012f-4f23-b573-c8c03fbf0628" containerName="cleanup"
Jul 06 15:48:18.952358 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.952260    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd7d8276-97fb-449c-91e1-1a014868a46b" containerName="cleanup"
Jul 06 15:48:18.955202 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.955186    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:18.957923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.957901    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6bb1764c\"/\"default-dockercfg-fmc4d\""
Jul 06 15:48:18.958212 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.957966    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6bb1764c\"/\"kube-root-ca.crt\""
Jul 06 15:48:18.958755 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.958735    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6bb1764c\"/\"openshift-service-ca.crt\""
Jul 06 15:48:18.958885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.958867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6bb1764c\"/\"e2e-lifecycle-model-6bb1764c-kserve-self-signed-certs\""
Jul 06 15:48:18.966107 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:18.966083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:48:19.025600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.025600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.025793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.025793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025684    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.025793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hjkj\" (UniqueName: \"kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.025793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.025747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.126916 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.126881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.126923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.126950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.126975    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.127014    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127098 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.127052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hjkj\" (UniqueName: \"kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127435 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.127384    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127507 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.127421    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.127507 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.127495    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.129285 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.129264    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.129550 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.129534    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.136441 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.136419    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hjkj\" (UniqueName: \"kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj\") pod \"e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") " pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.265280 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.265184    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:19.418814 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.418789    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:48:19.421375 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:48:19.421350    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a56597_2915_4436_ae2d_7b80fb5e9043.slice/crio-8032f000f53689ff075a4dcb6c7a87e3901c015d22a368db060356e06bf457db WatchSource:0}: Error finding container 8032f000f53689ff075a4dcb6c7a87e3901c015d22a368db060356e06bf457db: Status 404 returned error can't find the container with id 8032f000f53689ff075a4dcb6c7a87e3901c015d22a368db060356e06bf457db
Jul 06 15:48:19.478518 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:19.478490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" event={"ID":"16a56597-2915-4436-ae2d-7b80fb5e9043","Type":"ContainerStarted","Data":"8032f000f53689ff075a4dcb6c7a87e3901c015d22a368db060356e06bf457db"}
Jul 06 15:48:20.485856 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:20.485817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" event={"ID":"16a56597-2915-4436-ae2d-7b80fb5e9043","Type":"ContainerStarted","Data":"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"}
Jul 06 15:48:20.507611 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:20.507564    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" podStartSLOduration=2.507549042 podStartE2EDuration="2.507549042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:48:20.506442331 +0000 UTC m=+1881.286871154" watchObservedRunningTime="2026-07-06 15:48:20.507549042 +0000 UTC m=+1881.287977893"
Jul 06 15:48:22.453112 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:22.453028    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:48:29.266279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:29.266242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:29.266687 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:29.266311    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:29.278480 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:29.278457    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:29.542872 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:29.542791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:48:38.857015 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:38.856975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:48:52.693906 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:52.693864    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:48:52.844771 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:48:52.844741    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:49:07.714062 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:07.714005    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" podUID="16a56597-2915-4436-ae2d-7b80fb5e9043" containerName="main" containerID="cri-o://b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a" gracePeriod=45
Jul 06 15:49:07.963845 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:07.963812    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:49:08.147559 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147527    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.147737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147589    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hjkj\" (UniqueName: \"kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.147737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147610    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.147737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.147737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147674    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.147737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147705    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm\") pod \"16a56597-2915-4436-ae2d-7b80fb5e9043\" (UID: \"16a56597-2915-4436-ae2d-7b80fb5e9043\") "
Jul 06 15:49:08.148023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147862    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:49:08.148023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147907    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache" (OuterVolumeSpecName: "model-cache") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:49:08.148023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.147949    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home" (OuterVolumeSpecName: "home") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:49:08.149842 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.149814    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm" (OuterVolumeSpecName: "dshm") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:49:08.149946 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.149873    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:49:08.149946 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.149895    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj" (OuterVolumeSpecName: "kube-api-access-6hjkj") pod "16a56597-2915-4436-ae2d-7b80fb5e9043" (UID: "16a56597-2915-4436-ae2d-7b80fb5e9043"). InnerVolumeSpecName "kube-api-access-6hjkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:49:08.248765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248726    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.248765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248761    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hjkj\" (UniqueName: \"kubernetes.io/projected/16a56597-2915-4436-ae2d-7b80fb5e9043-kube-api-access-6hjkj\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.248765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248771    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/16a56597-2915-4436-ae2d-7b80fb5e9043-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.249006 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248780    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.249006 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248788    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.249006 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.248796    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/16a56597-2915-4436-ae2d-7b80fb5e9043-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:49:08.728477 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.728438    2566 generic.go:356] "Generic (PLEG): container finished" podID="16a56597-2915-4436-ae2d-7b80fb5e9043" containerID="b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a" exitCode=0
Jul 06 15:49:08.728947 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.728485    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"
Jul 06 15:49:08.728947 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.728527    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" event={"ID":"16a56597-2915-4436-ae2d-7b80fb5e9043","Type":"ContainerDied","Data":"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"}
Jul 06 15:49:08.728947 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.728586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb" event={"ID":"16a56597-2915-4436-ae2d-7b80fb5e9043","Type":"ContainerDied","Data":"8032f000f53689ff075a4dcb6c7a87e3901c015d22a368db060356e06bf457db"}
Jul 06 15:49:08.728947 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.728613    2566 scope.go:117] "RemoveContainer" containerID="b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"
Jul 06 15:49:08.737801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.737785    2566 scope.go:117] "RemoveContainer" containerID="b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"
Jul 06 15:49:08.738030 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:49:08.738014    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a\": container with ID starting with b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a not found: ID does not exist" containerID="b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"
Jul 06 15:49:08.738083 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.738038    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a"} err="failed to get container status \"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a\": rpc error: code = NotFound desc = could not find container \"b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a\": container with ID starting with b523d35fb31440a72a6b3af3ed98535553215720212afe8e450c5bc14439709a not found: ID does not exist"
Jul 06 15:49:08.750031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.750003    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:49:08.754901 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:08.754879    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6bb1764c/e2e-lifecycle-model-6bb1764c-kserve-5f8c8d4f9-8t9bb"]
Jul 06 15:49:09.721517 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:09.721482    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16a56597-2915-4436-ae2d-7b80fb5e9043" path="/var/lib/kubelet/pods/16a56597-2915-4436-ae2d-7b80fb5e9043/volumes"
Jul 06 15:49:32.459568 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:32.459529    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:49:46.766917 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:46.766883    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:49:58.960615 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:49:58.960537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:50:49.492441 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.492381    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:50:49.492922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.492785    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16a56597-2915-4436-ae2d-7b80fb5e9043" containerName="main"
Jul 06 15:50:49.492922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.492798    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="16a56597-2915-4436-ae2d-7b80fb5e9043" containerName="main"
Jul 06 15:50:49.492922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.492868    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="16a56597-2915-4436-ae2d-7b80fb5e9043" containerName="main"
Jul 06 15:50:49.495719 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.495702    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.500279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.500256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 06 15:50:49.501720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.501698    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-edc987-openshift-default-dockercfg-b65d5\""
Jul 06 15:50:49.528445 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.528412    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:50:49.582463 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582419    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582499    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9ljz\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582580    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582628    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582787    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.582855 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.582836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684032 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.683997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684032 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684041    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684127    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684173    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9ljz\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684204    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684281    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684780 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684489    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684780 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684526    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684780 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684768    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.684998 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.684972    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.685045 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.685020    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.686891 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.686864    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.687133 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.687114    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.699405 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.699367    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9ljz\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.712006 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.711982    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token\") pod \"e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") " pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.805508 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.805419    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:49.976737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.976710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:50:49.977419 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:50:49.977380    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa2e250e_d1dc_4971_a2d2_1d783009c86c.slice/crio-9ac73e96a9bf8887e4b4997bfbee6acd3b42640ce26534a0c28eb17e033d8955 WatchSource:0}: Error finding container 9ac73e96a9bf8887e4b4997bfbee6acd3b42640ce26534a0c28eb17e033d8955: Status 404 returned error can't find the container with id 9ac73e96a9bf8887e4b4997bfbee6acd3b42640ce26534a0c28eb17e033d8955
Jul 06 15:50:49.979346 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:49.979331    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 15:50:50.220546 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:50.220511    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerStarted","Data":"9ac73e96a9bf8887e4b4997bfbee6acd3b42640ce26534a0c28eb17e033d8955"}
Jul 06 15:50:52.061717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:52.061681    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:50:54.243320 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:54.243275    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:50:54.243671 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:54.243350    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:50:54.243671 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:54.243381    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:50:55.254778 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:55.254745    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerStarted","Data":"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"}
Jul 06 15:50:55.280373 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:55.280313    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podStartSLOduration=2.016831667 podStartE2EDuration="6.280294056s" totalImagesPullingTime="4.263462389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:50:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:50:55.279414797 +0000 UTC m=+2036.059843605" watchObservedRunningTime="2026-07-06 15:50:55.280294056 +0000 UTC m=+2036.060722858"
Jul 06 15:50:55.806372 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:55.806339    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:55.811316 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:55.811290    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:56.259759 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:56.259731    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:50:56.260657 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:50:56.260639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:51:03.006089 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:03.006052    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:51:11.261259 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.261219    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 06 15:51:11.261773 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.261282    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 06 15:51:11.343846 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.343805    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerID="0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a" exitCode=0
Jul 06 15:51:11.344031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.343884    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerDied","Data":"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"}
Jul 06 15:51:11.344346 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.344330    2566 scope.go:117] "RemoveContainer" containerID="0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"
Jul 06 15:51:11.345560 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.345521    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:51:11.345651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.345617    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:51:11.345690 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:11.345671    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:51:12.351038 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.351005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerStarted","Data":"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"}
Jul 06 15:51:12.679292 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.679210    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:51:12.689704 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.689669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.696173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.696139    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.696465 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.696446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.696635 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.696620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.696763 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.696746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.696905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.696892    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.697053 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.697034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.697997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.697966    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a30a8\"/\"default-dockercfg-6hjtn\""
Jul 06 15:51:12.700888 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.700868    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7a30a8\"/\"test-model-7a30a8-kserve-self-signed-certs\""
Jul 06 15:51:12.701369 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.701349    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a30a8\"/\"openshift-service-ca.crt\""
Jul 06 15:51:12.701576 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.701554    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7a30a8\"/\"kube-root-ca.crt\""
Jul 06 15:51:12.712604 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.712577    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:51:12.797699 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797655    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.797899 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.797899 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.797899 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797854    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.797899 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.798134 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.797919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.798663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.798637    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.799104 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.799084    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.799905 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.799868    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.803643 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.803598    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.808199 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.807888    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:51:12.808199 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.807989    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 06 15:51:12.808199 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.808038    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 06 15:51:12.809318 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.809283    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:12.813881 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:12.813853    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb\") pod \"test-model-7a30a8-kserve-7cb8b968f-99rlh\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") " pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:13.014056 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.013965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:13.197326 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.197287    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:51:13.216437 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:51:13.215762    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46f5e673_c07a_4de5_b0b1_485df0a19e51.slice/crio-57de36f7f72a07ef506b8cb45eaeee7a06dac87d31b091d8a0e9da60058c906b WatchSource:0}: Error finding container 57de36f7f72a07ef506b8cb45eaeee7a06dac87d31b091d8a0e9da60058c906b: Status 404 returned error can't find the container with id 57de36f7f72a07ef506b8cb45eaeee7a06dac87d31b091d8a0e9da60058c906b
Jul 06 15:51:13.358425 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.358374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" event={"ID":"46f5e673-c07a-4de5-b0b1-485df0a19e51","Type":"ContainerStarted","Data":"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"}
Jul 06 15:51:13.358810 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.358441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" event={"ID":"46f5e673-c07a-4de5-b0b1-485df0a19e51","Type":"ContainerStarted","Data":"57de36f7f72a07ef506b8cb45eaeee7a06dac87d31b091d8a0e9da60058c906b"}
Jul 06 15:51:13.386725 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.386658    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" podStartSLOduration=1.386635345 podStartE2EDuration="1.386635345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:51:13.383991864 +0000 UTC m=+2054.164420666" watchObservedRunningTime="2026-07-06 15:51:13.386635345 +0000 UTC m=+2054.167064148"
Jul 06 15:51:13.811437 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.809907    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 06 15:51:13.811437 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:13.809988    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 06 15:51:14.806676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:14.806629    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 06 15:51:14.807188 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:14.806700    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 06 15:51:15.806141 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:15.806090    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Jul 06 15:51:15.806364 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:15.806163    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Jul 06 15:51:16.809551 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:16.809521    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:51:16.809910 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:16.809791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:51:16.810516 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:16.810498    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:51:17.863422 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:17.863377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:51:23.014347 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:23.014254    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:23.014347 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:23.014313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:23.027253 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:23.027225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:23.426238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:23.426204    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:51:28.392997 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:28.392962    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:51:30.276283 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.276250    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:51:30.283084 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.283058    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.285992 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.285959    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-edc987\"/\"openshift-service-ca.crt\""
Jul 06 15:51:30.286136 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.286116    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-edc987\"/\"kube-root-ca.crt\""
Jul 06 15:51:30.286922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.286905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-edc987\"/\"test-model-edc987-kserve-self-signed-certs\""
Jul 06 15:51:30.287030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.286940    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-edc987\"/\"default-dockercfg-bjzgp\""
Jul 06 15:51:30.296050 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.296026    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:51:30.335883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.335854    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.336023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.335901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.336023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.335928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.336023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.335955    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lzww\" (UniqueName: \"kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.336023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.335987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.336023 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.336014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lzww\" (UniqueName: \"kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436776 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436743    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.436999 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.436794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.437112 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.437012    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.437177 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.437137    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.437326 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.437268    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.439454 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.439409    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.439724 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.439705    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.456585 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.456563    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lzww\" (UniqueName: \"kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww\") pod \"test-model-edc987-kserve-5dfcbcb9d5-vbbnm\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") " pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.593440 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.593382    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:30.753291 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:30.753254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:51:30.761553 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:51:30.761526    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58c13404_b70d_4e1e_bd41_fbf2e20d2698.slice/crio-f16a542efb200cfeda79877fe8812007fbadf64373868629c57cbb836d90ebf8 WatchSource:0}: Error finding container f16a542efb200cfeda79877fe8812007fbadf64373868629c57cbb836d90ebf8: Status 404 returned error can't find the container with id f16a542efb200cfeda79877fe8812007fbadf64373868629c57cbb836d90ebf8
Jul 06 15:51:31.452578 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:31.452544    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" event={"ID":"58c13404-b70d-4e1e-bd41-fbf2e20d2698","Type":"ContainerStarted","Data":"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"}
Jul 06 15:51:31.452992 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:31.452586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" event={"ID":"58c13404-b70d-4e1e-bd41-fbf2e20d2698","Type":"ContainerStarted","Data":"f16a542efb200cfeda79877fe8812007fbadf64373868629c57cbb836d90ebf8"}
Jul 06 15:51:31.480084 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:31.480032    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" podStartSLOduration=1.4800173779999999 podStartE2EDuration="1.480017378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:51:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:51:31.476645889 +0000 UTC m=+2072.257074712" watchObservedRunningTime="2026-07-06 15:51:31.480017378 +0000 UTC m=+2072.260446178"
Jul 06 15:51:40.204542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:40.204506    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:51:40.593584 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:40.593548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:40.593768 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:40.593609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:40.607131 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:40.607097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:41.514313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:41.514282    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:51:50.349563 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:50.349524    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:51:51.344545 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:51.344502    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:51:55.662182 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:51:55.662136    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:52:05.369439 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.369364    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" podUID="46f5e673-c07a-4de5-b0b1-485df0a19e51" containerName="main" containerID="cri-o://ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6" gracePeriod=45
Jul 06 15:52:05.614887 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.614860    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:52:05.622046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.621983    2566 generic.go:356] "Generic (PLEG): container finished" podID="46f5e673-c07a-4de5-b0b1-485df0a19e51" containerID="ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6" exitCode=0
Jul 06 15:52:05.622046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.622015    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"
Jul 06 15:52:05.622183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.622060    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" event={"ID":"46f5e673-c07a-4de5-b0b1-485df0a19e51","Type":"ContainerDied","Data":"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"}
Jul 06 15:52:05.622183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.622101    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh" event={"ID":"46f5e673-c07a-4de5-b0b1-485df0a19e51","Type":"ContainerDied","Data":"57de36f7f72a07ef506b8cb45eaeee7a06dac87d31b091d8a0e9da60058c906b"}
Jul 06 15:52:05.622183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.622118    2566 scope.go:117] "RemoveContainer" containerID="ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"
Jul 06 15:52:05.633040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.633023    2566 scope.go:117] "RemoveContainer" containerID="ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"
Jul 06 15:52:05.633284 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:52:05.633264    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6\": container with ID starting with ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6 not found: ID does not exist" containerID="ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"
Jul 06 15:52:05.633356 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.633298    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6"} err="failed to get container status \"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6\": rpc error: code = NotFound desc = could not find container \"ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6\": container with ID starting with ddb993bc5e6128c083cd474dd5b863b44964cba5113f7030ffa6a89d2c9acbc6 not found: ID does not exist"
Jul 06 15:52:05.761837 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.761804    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.761975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.761862    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.761975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.761887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.761975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.761914    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.762115 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762057    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.762170 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762115    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm\") pod \"46f5e673-c07a-4de5-b0b1-485df0a19e51\" (UID: \"46f5e673-c07a-4de5-b0b1-485df0a19e51\") "
Jul 06 15:52:05.762311 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762110    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:05.762472 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762206    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache" (OuterVolumeSpecName: "model-cache") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:05.762472 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762342    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home" (OuterVolumeSpecName: "home") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:05.762572 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762492    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.762572 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762514    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.762572 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.762527    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.764102 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.764083    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm" (OuterVolumeSpecName: "dshm") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:05.764189 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.764163    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb" (OuterVolumeSpecName: "kube-api-access-8tdpb") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "kube-api-access-8tdpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:52:05.764256 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.764239    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46f5e673-c07a-4de5-b0b1-485df0a19e51" (UID: "46f5e673-c07a-4de5-b0b1-485df0a19e51"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:52:05.863638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.863602    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tdpb\" (UniqueName: \"kubernetes.io/projected/46f5e673-c07a-4de5-b0b1-485df0a19e51-kube-api-access-8tdpb\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.863638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.863630    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46f5e673-c07a-4de5-b0b1-485df0a19e51-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.863638 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.863641    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46f5e673-c07a-4de5-b0b1-485df0a19e51-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:05.946440 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.946408    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:52:05.950438 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:05.950411    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7a30a8/test-model-7a30a8-kserve-7cb8b968f-99rlh"]
Jul 06 15:52:06.363648 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.363615    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" podUID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" containerName="main" containerID="cri-o://56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f" gracePeriod=45
Jul 06 15:52:06.620165 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.620095    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:52:06.628807 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.628780    2566 generic.go:356] "Generic (PLEG): container finished" podID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" containerID="56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f" exitCode=0
Jul 06 15:52:06.628933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.628817    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"
Jul 06 15:52:06.628933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.628873    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" event={"ID":"58c13404-b70d-4e1e-bd41-fbf2e20d2698","Type":"ContainerDied","Data":"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"}
Jul 06 15:52:06.628933 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.628916    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm" event={"ID":"58c13404-b70d-4e1e-bd41-fbf2e20d2698","Type":"ContainerDied","Data":"f16a542efb200cfeda79877fe8812007fbadf64373868629c57cbb836d90ebf8"}
Jul 06 15:52:06.629065 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.628935    2566 scope.go:117] "RemoveContainer" containerID="56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"
Jul 06 15:52:06.636957 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.636942    2566 scope.go:117] "RemoveContainer" containerID="56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"
Jul 06 15:52:06.637184 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:52:06.637166    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f\": container with ID starting with 56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f not found: ID does not exist" containerID="56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"
Jul 06 15:52:06.637244 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.637192    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f"} err="failed to get container status \"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f\": rpc error: code = NotFound desc = could not find container \"56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f\": container with ID starting with 56c86ded8ed4f092dae51aa56da57870fcda87425bfd2825bc70db1d58386d1f not found: ID does not exist"
Jul 06 15:52:06.770711 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770678    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.770883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770727    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.770883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.770883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lzww\" (UniqueName: \"kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.770883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770880    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.771044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.770918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm\") pod \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\" (UID: \"58c13404-b70d-4e1e-bd41-fbf2e20d2698\") "
Jul 06 15:52:06.771044 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771021    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache" (OuterVolumeSpecName: "model-cache") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:06.771148 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771063    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:06.771203 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771160    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home" (OuterVolumeSpecName: "home") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:06.771277 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771256    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.771277 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771269    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.771353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.771279    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.773036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.773015    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm" (OuterVolumeSpecName: "dshm") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:06.773145 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.773040    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww" (OuterVolumeSpecName: "kube-api-access-2lzww") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "kube-api-access-2lzww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:52:06.773145 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.773117    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "58c13404-b70d-4e1e-bd41-fbf2e20d2698" (UID: "58c13404-b70d-4e1e-bd41-fbf2e20d2698"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:52:06.872153 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.872046    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lzww\" (UniqueName: \"kubernetes.io/projected/58c13404-b70d-4e1e-bd41-fbf2e20d2698-kube-api-access-2lzww\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.872153 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.872087    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/58c13404-b70d-4e1e-bd41-fbf2e20d2698-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.872153 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.872104    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/58c13404-b70d-4e1e-bd41-fbf2e20d2698-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:06.952746 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.952717    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:52:06.957205 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:06.957181    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-edc987/test-model-edc987-kserve-5dfcbcb9d5-vbbnm"]
Jul 06 15:52:07.721043 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:07.721009    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46f5e673-c07a-4de5-b0b1-485df0a19e51" path="/var/lib/kubelet/pods/46f5e673-c07a-4de5-b0b1-485df0a19e51/volumes"
Jul 06 15:52:07.721460 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:07.721445    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" path="/var/lib/kubelet/pods/58c13404-b70d-4e1e-bd41-fbf2e20d2698/volumes"
Jul 06 15:52:39.917229 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:39.917191    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:52:43.754221 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:43.754182    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:52:43.754742 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:43.754461    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" containerID="cri-o://f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3" gracePeriod=30
Jul 06 15:52:44.659663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.659630    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:52:44.659974 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.659963    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46f5e673-c07a-4de5-b0b1-485df0a19e51" containerName="main"
Jul 06 15:52:44.660029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.659976    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46f5e673-c07a-4de5-b0b1-485df0a19e51" containerName="main"
Jul 06 15:52:44.660029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.659997    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" containerName="main"
Jul 06 15:52:44.660029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.660003    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" containerName="main"
Jul 06 15:52:44.660157 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.660057    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46f5e673-c07a-4de5-b0b1-485df0a19e51" containerName="main"
Jul 06 15:52:44.660157 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.660064    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58c13404-b70d-4e1e-bd41-fbf2e20d2698" containerName="main"
Jul 06 15:52:44.663282 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.663263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.666061 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.666039    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-7f81c0-openshift-default-dockercfg-x9mqv\""
Jul 06 15:52:44.678932 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.678909    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:52:44.767333 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767301    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767460    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767535    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767642    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.767718 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.767657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwdhv\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.868652 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.868818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.868818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.868818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.868818 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869014 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868833    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869014 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869014 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwdhv\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869014 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.868961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869223 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.869203    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869347 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.869318    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.869330    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.869462    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.869511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.869504    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.871121 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.871097    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.871217 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.871192    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.883256 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.883230    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.887237 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.887210    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwdhv\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv\") pod \"e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") " pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:44.973209 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:44.973133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:45.119836 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.119810    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:52:45.121360 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:52:45.121328    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9383d334_2c90_4916_9cb6_cc06a340cd87.slice/crio-6ce6671a678bcbb44526fa782a3069d6ce5161d65b19a0139b1c7545f3ae40e1 WatchSource:0}: Error finding container 6ce6671a678bcbb44526fa782a3069d6ce5161d65b19a0139b1c7545f3ae40e1: Status 404 returned error can't find the container with id 6ce6671a678bcbb44526fa782a3069d6ce5161d65b19a0139b1c7545f3ae40e1
Jul 06 15:52:45.123651 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.123617    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:45.123717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.123691    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:45.123761 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.123722    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:45.843031 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.842996    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerStarted","Data":"2672a4f1674713ca32e18b183a29a1fd9862f8abf1e62db91b8bf271c8406685"}
Jul 06 15:52:45.843571 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.843083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerStarted","Data":"6ce6671a678bcbb44526fa782a3069d6ce5161d65b19a0139b1c7545f3ae40e1"}
Jul 06 15:52:45.873952 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.873904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podStartSLOduration=1.873890804 podStartE2EDuration="1.873890804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:52:45.871814153 +0000 UTC m=+2146.652242988" watchObservedRunningTime="2026-07-06 15:52:45.873890804 +0000 UTC m=+2146.654319605"
Jul 06 15:52:45.973948 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:45.973916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:46.811891 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:46.811855    2566 patch_prober.go:28] interesting pod/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:52:46.812074 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:46.811917    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:52:46.979005 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:46.978974    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:47.853759 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:47.853720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:47.854814 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:47.854792    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:52:49.002581 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.002559    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:52:49.109556 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109524    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h9ljz\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109603    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109632    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.109733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109720    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.110059 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.110059 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109811    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo\") pod \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\" (UID: \"aa2e250e-d1dc-4971-a2d2-1d783009c86c\") "
Jul 06 15:52:49.110059 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.109956    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:49.110238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.110071    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.110238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.110082    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:52:49.110238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.110223    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:49.110793 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.110765    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data" (OuterVolumeSpecName: "istio-data") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:49.110912 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.110839    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:49.111885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.111856    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token" (OuterVolumeSpecName: "istio-token") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:52:49.111989 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.111911    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz" (OuterVolumeSpecName: "kube-api-access-h9ljz") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "kube-api-access-h9ljz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:52:49.112036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.112011    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 15:52:49.112036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.112014    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aa2e250e-d1dc-4971-a2d2-1d783009c86c" (UID: "aa2e250e-d1dc-4971-a2d2-1d783009c86c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:52:49.211187 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211151    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-workload-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211187 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211181    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-podinfo\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211187 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211189    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211199    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9ljz\" (UniqueName: \"kubernetes.io/projected/aa2e250e-d1dc-4971-a2d2-1d783009c86c-kube-api-access-h9ljz\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211209    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-credential-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211218    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-envoy\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211271    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istiod-ca-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.211449 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.211281    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa2e250e-d1dc-4971-a2d2-1d783009c86c-istio-data\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:52:49.864313 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.864277    2566 generic.go:356] "Generic (PLEG): container finished" podID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerID="f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3" exitCode=0
Jul 06 15:52:49.864511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.864326    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"
Jul 06 15:52:49.864511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.864365    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerDied","Data":"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"}
Jul 06 15:52:49.864511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.864445    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f" event={"ID":"aa2e250e-d1dc-4971-a2d2-1d783009c86c","Type":"ContainerDied","Data":"9ac73e96a9bf8887e4b4997bfbee6acd3b42640ce26534a0c28eb17e033d8955"}
Jul 06 15:52:49.864511 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.864469    2566 scope.go:117] "RemoveContainer" containerID="f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"
Jul 06 15:52:49.873684 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.873488    2566 scope.go:117] "RemoveContainer" containerID="0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"
Jul 06 15:52:49.881213 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.881197    2566 scope.go:117] "RemoveContainer" containerID="f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"
Jul 06 15:52:49.881502 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:52:49.881476    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3\": container with ID starting with f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3 not found: ID does not exist" containerID="f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"
Jul 06 15:52:49.881595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.881514    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3"} err="failed to get container status \"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3\": rpc error: code = NotFound desc = could not find container \"f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3\": container with ID starting with f1ff55576c294410d2dffaa5e140f1ebc0a89471d5f6f65422957a55c35d20e3 not found: ID does not exist"
Jul 06 15:52:49.881595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.881531    2566 scope.go:117] "RemoveContainer" containerID="0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"
Jul 06 15:52:49.881793 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:52:49.881772    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a\": container with ID starting with 0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a not found: ID does not exist" containerID="0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"
Jul 06 15:52:49.881853 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.881799    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a"} err="failed to get container status \"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a\": rpc error: code = NotFound desc = could not find container \"0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a\": container with ID starting with 0cfdacac67e9673d5d74a7db35417d59e24c1ebc815fe1ec7ee9a319ada9c02a not found: ID does not exist"
Jul 06 15:52:49.898887 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.898863    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:52:49.907370 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:49.907351    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-edc987-openshift-default-7bfcf99cff-f825f"]
Jul 06 15:52:51.721531 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:51.721498    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" path="/var/lib/kubelet/pods/aa2e250e-d1dc-4971-a2d2-1d783009c86c/volumes"
Jul 06 15:52:53.223063 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.222978    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:52:53.223453 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223353    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.223453 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223365    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.223453 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223373    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.223453 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223379    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.223589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223478    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.223589 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.223487    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa2e250e-d1dc-4971-a2d2-1d783009c86c" containerName="istio-proxy"
Jul 06 15:52:53.227886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.227867    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.231975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.231955    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c967ab-openshift-default-dockercfg-ddvkr\""
Jul 06 15:52:53.246710 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.246688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:52:53.351164 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351177    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351197    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351318    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351348 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351372    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.351600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.351423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcr5r\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452344    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452381 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452531    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452663 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452659    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452685    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.452975 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.452846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcr5r\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.453056 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.453033    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.453136 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.453111    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.453214 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.453169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.453372 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.453354    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.453444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.453375    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.454822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.454805    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.455052 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.455033    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.462266 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.462240    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcr5r\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.462706 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.462679    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token\") pod \"e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") " pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.537843 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.537770    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:53.654809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.654772    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:52:53.686004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.685969    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:52:53.688450 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:52:53.688416    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3eb9c3f_34bb_46e3_86e3_84c3b293be19.slice/crio-f188c71b835b090b0eb6d5f9eef6cc27c716f12db386a742eee5553ca60a8a75 WatchSource:0}: Error finding container f188c71b835b090b0eb6d5f9eef6cc27c716f12db386a742eee5553ca60a8a75: Status 404 returned error can't find the container with id f188c71b835b090b0eb6d5f9eef6cc27c716f12db386a742eee5553ca60a8a75
Jul 06 15:52:53.690846 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.690806    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:53.690926 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.690902    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:53.690972 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.690947    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:52:53.894785 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.894743    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerStarted","Data":"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"}
Jul 06 15:52:53.894999 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.894812    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerStarted","Data":"f188c71b835b090b0eb6d5f9eef6cc27c716f12db386a742eee5553ca60a8a75"}
Jul 06 15:52:53.924796 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:53.924739    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podStartSLOduration=0.924724192 podStartE2EDuration="924.724192ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:52:53.921570559 +0000 UTC m=+2154.701999371" watchObservedRunningTime="2026-07-06 15:52:53.924724192 +0000 UTC m=+2154.705152996"
Jul 06 15:52:54.538349 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:54.538313    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:54.543764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:54.543735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:54.902667 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:54.902642    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:52:54.903507 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:52:54.903487    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:02.855211 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.855173    2566 patch_prober.go:28] interesting pod/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 06 15:53:02.855662 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.855227    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 15:53:02.947236 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.947201    2566 generic.go:356] "Generic (PLEG): container finished" podID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerID="2672a4f1674713ca32e18b183a29a1fd9862f8abf1e62db91b8bf271c8406685" exitCode=0
Jul 06 15:53:02.947427 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.947275    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerDied","Data":"2672a4f1674713ca32e18b183a29a1fd9862f8abf1e62db91b8bf271c8406685"}
Jul 06 15:53:02.947682 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.947669    2566 scope.go:117] "RemoveContainer" containerID="2672a4f1674713ca32e18b183a29a1fd9862f8abf1e62db91b8bf271c8406685"
Jul 06 15:53:02.948764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.948730    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:02.948840 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.948804    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:02.948840 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:02.948836    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:03.954816 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:03.954777    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerStarted","Data":"cd08c64f87b90b9c00f10d9e559a3aafd5a0f107dfbe2aaef51db4e1e1a1dc87"}
Jul 06 15:53:03.973501 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:03.973469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:04.974466 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:04.974432    2566 patch_prober.go:28] interesting pod/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 06 15:53:04.974863 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:04.974482    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 15:53:05.974147 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:05.974116    2566 patch_prober.go:28] interesting pod/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 06 15:53:05.974414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:05.974366    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 15:53:06.974476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:06.974442    2566 patch_prober.go:28] interesting pod/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 06 15:53:06.974952 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:06.974500    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 15:53:07.974230 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:07.974184    2566 patch_prober.go:28] interesting pod/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Jul 06 15:53:07.974429 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:07.974249    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Jul 06 15:53:08.977696 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.977665    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:08.978173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.978151    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:08.978895 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.978878    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:08.989007 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.988974    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerID="84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22" exitCode=0
Jul 06 15:53:08.989159 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.989045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerDied","Data":"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"}
Jul 06 15:53:08.989837 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.989817    2566 scope.go:117] "RemoveContainer" containerID="84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"
Jul 06 15:53:08.990806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.990779    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:08.990896 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.990849    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:08.990896 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:08.990880    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:09.902520 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:09.902481    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:09.930765 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:09.930733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:53:10.002303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:10.002260    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerStarted","Data":"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"}
Jul 06 15:53:10.538200 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:10.538160    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:10.539464 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:10.539437    2566 patch_prober.go:28] interesting pod/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 06 15:53:10.539583 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:10.539490    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 06 15:53:11.538839 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:11.538801    2566 patch_prober.go:28] interesting pod/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 06 15:53:11.539226 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:11.538857    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 06 15:53:12.538165 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:12.538128    2566 patch_prober.go:28] interesting pod/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 06 15:53:12.538418 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:12.538180    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 06 15:53:13.538571 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:13.538529    2566 patch_prober.go:28] interesting pod/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Jul 06 15:53:13.538955 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:13.538600    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Jul 06 15:53:14.542780 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:14.542749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:14.543152 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:14.543132    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:14.543862 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:14.543840    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:16.831317 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:16.831286    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:53:16.831736 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:16.831496    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy" containerID="cri-o://cd08c64f87b90b9c00f10d9e559a3aafd5a0f107dfbe2aaef51db4e1e1a1dc87" gracePeriod=30
Jul 06 15:53:22.068962 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.068930    2566 generic.go:356] "Generic (PLEG): container finished" podID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerID="cd08c64f87b90b9c00f10d9e559a3aafd5a0f107dfbe2aaef51db4e1e1a1dc87" exitCode=0
Jul 06 15:53:22.069424 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.068999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerDied","Data":"cd08c64f87b90b9c00f10d9e559a3aafd5a0f107dfbe2aaef51db4e1e1a1dc87"}
Jul 06 15:53:22.069424 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.069063    2566 scope.go:117] "RemoveContainer" containerID="2672a4f1674713ca32e18b183a29a1fd9862f8abf1e62db91b8bf271c8406685"
Jul 06 15:53:22.103078 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.103020    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:22.130632 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.130600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:53:22.204133 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204133 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204134    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwdhv\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204179    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204247    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204289    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204386 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204315    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204632 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204387    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:22.204632 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204420    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204632 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204632 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204510    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token\") pod \"9383d334-2c90-4916-9cb6-cc06a340cd87\" (UID: \"9383d334-2c90-4916-9cb6-cc06a340cd87\") "
Jul 06 15:53:22.204838 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204699    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:22.204838 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204786    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:53:22.204944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204924    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-credential-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.205004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204945    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9383d334-2c90-4916-9cb6-cc06a340cd87-istiod-ca-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.205004 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.204961    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.205273 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.205214    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data" (OuterVolumeSpecName: "istio-data") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:22.205359 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.205346    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:22.206908 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.206874    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv" (OuterVolumeSpecName: "kube-api-access-cwdhv") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "kube-api-access-cwdhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:53:22.207113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.206983    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:22.207113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.207057    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 15:53:22.207210 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.207119    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token" (OuterVolumeSpecName: "istio-token") pod "9383d334-2c90-4916-9cb6-cc06a340cd87" (UID: "9383d334-2c90-4916-9cb6-cc06a340cd87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:53:22.306076 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306048    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwdhv\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-kube-api-access-cwdhv\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.306076 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306077    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-podinfo\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.306299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306088    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-data\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.306299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306096    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-workload-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.306299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306103    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-envoy\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:22.306299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:22.306113    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9383d334-2c90-4916-9cb6-cc06a340cd87-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:23.076279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.076244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb" event={"ID":"9383d334-2c90-4916-9cb6-cc06a340cd87","Type":"ContainerDied","Data":"6ce6671a678bcbb44526fa782a3069d6ce5161d65b19a0139b1c7545f3ae40e1"}
Jul 06 15:53:23.076720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.076307    2566 scope.go:117] "RemoveContainer" containerID="cd08c64f87b90b9c00f10d9e559a3aafd5a0f107dfbe2aaef51db4e1e1a1dc87"
Jul 06 15:53:23.076720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.076262    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"
Jul 06 15:53:23.108571 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.108542    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:53:23.120361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.120332    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-7f81c0-openshift-default-676b89f69c-zf9sb"]
Jul 06 15:53:23.721253 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:23.721221    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" path="/var/lib/kubelet/pods/9383d334-2c90-4916-9cb6-cc06a340cd87/volumes"
Jul 06 15:53:25.437859 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:25.437819    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:53:25.438334 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:25.438072    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" containerID="cri-o://f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd" gracePeriod=30
Jul 06 15:53:26.460016 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.459976    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:53:26.460414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460313    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.460414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460325    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.460414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460340    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.460414 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460346    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.460565 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460425    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.460565 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.460432    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9383d334-2c90-4916-9cb6-cc06a340cd87" containerName="istio-proxy"
Jul 06 15:53:26.464835 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.464817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.480321 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.480298    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-379916-openshift-default-dockercfg-wq2fz\""
Jul 06 15:53:26.484713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.484688    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:53:26.544347 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485dc\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544368    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544543    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544577    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544592    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544634 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.544758 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.544637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645617 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485dc\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645672    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645717    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.645809 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646072 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645876    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646072 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646178 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.646107    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.646178    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646346 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.646319    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646499 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.646374    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646499 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.645986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.646697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.646679    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.647951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.647926    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.648467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.648445    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.661460 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.661431    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.661811 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.661789    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-485dc\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc\") pod \"e2e-shared-a-379916-openshift-default-b9f97969f-hfr86\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") " pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.774351 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.774268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:26.941403 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.941362    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:53:26.947626 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.947592    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:26.947728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.947656    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:26.947728 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:26.947686    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:27.101147 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:27.101117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerStarted","Data":"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"}
Jul 06 15:53:27.101331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:27.101172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerStarted","Data":"015a4c46fd9c9be634b4f547c7e6899fd4b3b386df9e02130b9713f44812dd36"}
Jul 06 15:53:27.154041 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:27.153997    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podStartSLOduration=1.15397887 podStartE2EDuration="1.15397887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:53:27.15319968 +0000 UTC m=+2187.933628503" watchObservedRunningTime="2026-07-06 15:53:27.15397887 +0000 UTC m=+2187.934407678"
Jul 06 15:53:27.774713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:27.774682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:27.779654 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:27.779631    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:28.105655 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:28.105624    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:28.106656 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:28.106640    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:29.545677 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:29.545643    2566 patch_prober.go:28] interesting pod/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:53:29.546045 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:29.545698    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:53:30.699300 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.699225    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:30.782915 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.782879    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.782927    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.782956    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.782988    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783275 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783275 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783267    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783378 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783297    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783378 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783357    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.783513 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783388    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcr5r\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r\") pod \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\" (UID: \"c3eb9c3f-34bb-46e3-86e3-84c3b293be19\") "
Jul 06 15:53:30.784574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.783968    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data" (OuterVolumeSpecName: "istio-data") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:30.784574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.784221    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:53:30.784574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.784472    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:30.784574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.784540    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:30.784886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.784644    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:30.786356 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.786329    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token" (OuterVolumeSpecName: "istio-token") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:53:30.786559 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.786447    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 15:53:30.786559 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.786551    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:53:30.786769 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.786740    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r" (OuterVolumeSpecName: "kube-api-access-kcr5r") pod "c3eb9c3f-34bb-46e3-86e3-84c3b293be19" (UID: "c3eb9c3f-34bb-46e3-86e3-84c3b293be19"). InnerVolumeSpecName "kube-api-access-kcr5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:53:30.885120 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885086    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885120 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885122    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istiod-ca-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885136    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-envoy\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885152    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcr5r\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-kube-api-access-kcr5r\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885166    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-podinfo\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885180    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885192    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-credential-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885204    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-istio-data\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:30.885361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:30.885216    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3eb9c3f-34bb-46e3-86e3-84c3b293be19-workload-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:53:31.122132 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.122096    2566 generic.go:356] "Generic (PLEG): container finished" podID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerID="f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd" exitCode=0
Jul 06 15:53:31.122303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.122143    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"
Jul 06 15:53:31.122303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.122189    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerDied","Data":"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"}
Jul 06 15:53:31.122303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.122244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t" event={"ID":"c3eb9c3f-34bb-46e3-86e3-84c3b293be19","Type":"ContainerDied","Data":"f188c71b835b090b0eb6d5f9eef6cc27c716f12db386a742eee5553ca60a8a75"}
Jul 06 15:53:31.122303 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.122267    2566 scope.go:117] "RemoveContainer" containerID="f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"
Jul 06 15:53:31.132085 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.132069    2566 scope.go:117] "RemoveContainer" containerID="84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"
Jul 06 15:53:31.139694 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.139677    2566 scope.go:117] "RemoveContainer" containerID="f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"
Jul 06 15:53:31.139931 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:53:31.139910    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd\": container with ID starting with f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd not found: ID does not exist" containerID="f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"
Jul 06 15:53:31.139986 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.139940    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd"} err="failed to get container status \"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd\": rpc error: code = NotFound desc = could not find container \"f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd\": container with ID starting with f9cc8314911788515fd395ebce6b9c1519f453c466d7ca9488783750c8c696fd not found: ID does not exist"
Jul 06 15:53:31.139986 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.139957    2566 scope.go:117] "RemoveContainer" containerID="84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"
Jul 06 15:53:31.140187 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:53:31.140172    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22\": container with ID starting with 84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22 not found: ID does not exist" containerID="84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"
Jul 06 15:53:31.140239 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.140197    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22"} err="failed to get container status \"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22\": rpc error: code = NotFound desc = could not find container \"84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22\": container with ID starting with 84139c279f0d0c41894915c3eab0b43c21b43cac68fb795c53fbe5e4536beb22 not found: ID does not exist"
Jul 06 15:53:31.156020 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.155998    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:53:31.168604 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.168581    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c967ab-openshift-default-55db7c8845-fnw6t"]
Jul 06 15:53:31.721768 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:31.721733    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" path="/var/lib/kubelet/pods/c3eb9c3f-34bb-46e3-86e3-84c3b293be19/volumes"
Jul 06 15:53:34.568447 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.568414    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:53:34.568886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.568738    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:34.568886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.568749    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:34.568886 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.568816    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:34.573280 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.573263    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.580046 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.580024    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-650de1-openshift-default-dockercfg-bb8qw\""
Jul 06 15:53:34.606555 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.606525    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:53:34.619619 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619746 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619633    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619746 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619660    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619820 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619820 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619808    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619888 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619888 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619952 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.619952 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.619936    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fddh\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720562 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720528    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720751 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720751 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720709    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.720876 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720846    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fddh\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720955    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721036 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.720999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.721043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.721161    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721302 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.721212    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721427 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.721385    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.721813 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.721791    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.723019 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.722994    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.723255 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.723234    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.730160 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.730139    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.730413 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.730371    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fddh\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh\") pod \"e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") " pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:34.883880 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:34.883826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:35.035595 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.035557    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:53:35.041058 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:53:35.041022    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd5c4fd_69be_4f92_9967_ec07284f186d.slice/crio-f850bf4e218a4d5c47efb893a3173e2d89b820965be7ffa40c3199f3a27f4b0f WatchSource:0}: Error finding container f850bf4e218a4d5c47efb893a3173e2d89b820965be7ffa40c3199f3a27f4b0f: Status 404 returned error can't find the container with id f850bf4e218a4d5c47efb893a3173e2d89b820965be7ffa40c3199f3a27f4b0f
Jul 06 15:53:35.043179 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.043148    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:35.043245 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.043215    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:35.043282 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.043245    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:35.145201 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.145113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerStarted","Data":"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"}
Jul 06 15:53:35.145201 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.145164    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerStarted","Data":"f850bf4e218a4d5c47efb893a3173e2d89b820965be7ffa40c3199f3a27f4b0f"}
Jul 06 15:53:35.178976 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.178916    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podStartSLOduration=1.178896401 podStartE2EDuration="1.178896401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:53:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:53:35.177474017 +0000 UTC m=+2195.957902821" watchObservedRunningTime="2026-07-06 15:53:35.178896401 +0000 UTC m=+2195.959325203"
Jul 06 15:53:35.396114 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.396032    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:53:35.884819 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:35.884791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:36.898205 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:36.897946    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:37.155469 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:37.155357    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:37.156470 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:37.156447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:43.132999 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.132640    2566 patch_prober.go:28] interesting pod/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:59846->10.132.0.54:15021: read: connection reset by peer" start-of-body=
Jul 06 15:53:43.132999 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.132714    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": read tcp 10.132.0.2:59846->10.132.0.54:15021: read: connection reset by peer"
Jul 06 15:53:43.192569 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.191560    2566 generic.go:356] "Generic (PLEG): container finished" podID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerID="63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67" exitCode=0
Jul 06 15:53:43.192569 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.191676    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerDied","Data":"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"}
Jul 06 15:53:43.192569 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.192172    2566 scope.go:117] "RemoveContainer" containerID="63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"
Jul 06 15:53:43.194138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.193789    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:43.194138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.193888    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:43.194138 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:43.193934    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:44.201574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:44.201516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerStarted","Data":"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"}
Jul 06 15:53:44.774627 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:44.774590    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:44.774922 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:44.774902    2566 patch_prober.go:28] interesting pod/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 06 15:53:44.774980 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:44.774945    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 06 15:53:45.775212 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:45.775178    2566 patch_prober.go:28] interesting pod/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 06 15:53:45.775623 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:45.775235    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 06 15:53:46.774740 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:46.774710    2566 patch_prober.go:28] interesting pod/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 06 15:53:46.774892 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:46.774759    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 06 15:53:47.774822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:47.774779    2566 patch_prober.go:28] interesting pod/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Jul 06 15:53:47.775327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:47.774837    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Jul 06 15:53:48.778226 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:48.778194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:48.778627 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:48.778469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:48.779099 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:48.779083    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:53:50.238751 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.238664    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerID="9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193" exitCode=0
Jul 06 15:53:50.239258 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.238741    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerDied","Data":"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"}
Jul 06 15:53:50.239258 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.239160    2566 scope.go:117] "RemoveContainer" containerID="9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"
Jul 06 15:53:50.240139 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.240109    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:50.240208 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.240190    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:50.240245 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.240224    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 06 15:53:50.456636 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:50.456597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:53:51.250943 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:51.250905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerStarted","Data":"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"}
Jul 06 15:53:51.884235 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:51.884203    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:51.885560 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:51.885522    2566 patch_prober.go:28] interesting pod/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 06 15:53:51.885704 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:51.885606    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 06 15:53:52.156327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:52.156237    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:52.884771 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:52.884727    2566 patch_prober.go:28] interesting pod/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 06 15:53:52.885260 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:52.884801    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 06 15:53:53.885041 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:53.885003    2566 patch_prober.go:28] interesting pod/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 06 15:53:53.885650 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:53.885615    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 06 15:53:54.884498 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:54.884459    2566 patch_prober.go:28] interesting pod/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Jul 06 15:53:54.884799 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:54.884767    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Jul 06 15:53:55.888692 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:55.888658    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:55.889693 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:55.889671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:53:58.670502 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.670469    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:53:58.670875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.670827    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:58.670875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.670837    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:58.670951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.670893    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3eb9c3f-34bb-46e3-86e3-84c3b293be19" containerName="istio-proxy"
Jul 06 15:53:58.674279 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.674261    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.677650 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.677617    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"default-dockercfg-bvmvm\""
Jul 06 15:53:58.677650 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.677628    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"kube-root-ca.crt\""
Jul 06 15:53:58.677856 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.677690    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"openshift-service-ca.crt\""
Jul 06 15:53:58.677856 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.677720    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"auth-test-model-379916-kserve-self-signed-certs\""
Jul 06 15:53:58.686971 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.686950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:53:58.851717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm7mm\" (UniqueName: \"kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.851717 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.851983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.851983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851826    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.851983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851860    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.851983 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.851881    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953268    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm7mm\" (UniqueName: \"kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953331    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953619 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953619 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953619 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953763 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953743    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953814 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953796    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.953875 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.953855    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.955679 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.955663    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.955854 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.955836    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.963442 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.963418    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm7mm\" (UniqueName: \"kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm\") pod \"auth-test-model-379916-kserve-65bb7d96ff-dwn2m\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") " pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:58.983379 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:58.983357    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:53:59.126083 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:59.126059    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:53:59.128349 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:53:59.128320    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode39487e3_8144_42c8_8c21_2ffa91e7b5bb.slice/crio-d1e2e22058e2e81cc5e4f5fec5413810299bc8640415a0fe26fb0c54885b2354 WatchSource:0}: Error finding container d1e2e22058e2e81cc5e4f5fec5413810299bc8640415a0fe26fb0c54885b2354: Status 404 returned error can't find the container with id d1e2e22058e2e81cc5e4f5fec5413810299bc8640415a0fe26fb0c54885b2354
Jul 06 15:53:59.294541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:59.294441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" event={"ID":"e39487e3-8144-42c8-8c21-2ffa91e7b5bb","Type":"ContainerStarted","Data":"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"}
Jul 06 15:53:59.294541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:59.294495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" event={"ID":"e39487e3-8144-42c8-8c21-2ffa91e7b5bb","Type":"ContainerStarted","Data":"d1e2e22058e2e81cc5e4f5fec5413810299bc8640415a0fe26fb0c54885b2354"}
Jul 06 15:53:59.330305 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:53:59.330252    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" podStartSLOduration=1.330235488 podStartE2EDuration="1.330235488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:53:59.327775221 +0000 UTC m=+2220.108204040" watchObservedRunningTime="2026-07-06 15:53:59.330235488 +0000 UTC m=+2220.110664289"
Jul 06 15:54:02.262801 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:02.262762    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:54:08.984476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:08.984430    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:54:08.984966 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:08.984493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:54:08.997607 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:08.997572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:54:09.356600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:09.356569    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:54:16.241191 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.241155    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:54:16.244553 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.244536    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.250213 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.250192    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"kube-root-ca.crt\""
Jul 06 15:54:16.251631 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.251310    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"default-dockercfg-zpztf\""
Jul 06 15:54:16.251631 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.251542    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"auth-test-model-650de1-kserve-self-signed-certs\""
Jul 06 15:54:16.252478 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.252459    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"openshift-service-ca.crt\""
Jul 06 15:54:16.275174 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.275147    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:54:16.310417 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.310596 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310493    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.310596 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310536    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.310596 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310563    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm4vd\" (UniqueName: \"kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.310771 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310613    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.310771 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.310699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412037 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412037 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm4vd\" (UniqueName: \"kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412309 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412309 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412309 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412309 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412540 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412521    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412639 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412616    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.412676 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.412629    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.414545 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.414524    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.414836 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.414814    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.427633 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.427605    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm4vd\" (UniqueName: \"kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd\") pod \"auth-test-model-650de1-kserve-65c84d44f5-mqwcc\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") " pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.554168 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.554074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:16.714051 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:16.714010    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:54:16.719660 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:54:16.719632    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e1b8798_c2bc_414f_8148_4dc1f9c54b05.slice/crio-c7cb729da989461dce36d4e063ff90f1a944052dfc57b7d94707dcbee2305a60 WatchSource:0}: Error finding container c7cb729da989461dce36d4e063ff90f1a944052dfc57b7d94707dcbee2305a60: Status 404 returned error can't find the container with id c7cb729da989461dce36d4e063ff90f1a944052dfc57b7d94707dcbee2305a60
Jul 06 15:54:17.387920 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:17.387886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" event={"ID":"4e1b8798-c2bc-414f-8148-4dc1f9c54b05","Type":"ContainerStarted","Data":"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"}
Jul 06 15:54:17.388371 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:17.387934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" event={"ID":"4e1b8798-c2bc-414f-8148-4dc1f9c54b05","Type":"ContainerStarted","Data":"c7cb729da989461dce36d4e063ff90f1a944052dfc57b7d94707dcbee2305a60"}
Jul 06 15:54:17.438822 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:17.438769    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" podStartSLOduration=1.438755018 podStartE2EDuration="1.438755018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:54:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:54:17.437370528 +0000 UTC m=+2238.217799329" watchObservedRunningTime="2026-07-06 15:54:17.438755018 +0000 UTC m=+2238.219183798"
Jul 06 15:54:19.776826 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:19.776794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:54:22.421353 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:22.421323    2566 generic.go:356] "Generic (PLEG): container finished" podID="5619760d-a735-4d20-9c8e-c83e5ea38129" containerID="54d81b6b140b9160bb6e7fc885f2c6a8890cde5aa1a36d79d8f8b76116257931" exitCode=1
Jul 06 15:54:22.421804 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:22.421424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp" event={"ID":"5619760d-a735-4d20-9c8e-c83e5ea38129","Type":"ContainerDied","Data":"54d81b6b140b9160bb6e7fc885f2c6a8890cde5aa1a36d79d8f8b76116257931"}
Jul 06 15:54:22.421804 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:22.421768    2566 scope.go:117] "RemoveContainer" containerID="54d81b6b140b9160bb6e7fc885f2c6a8890cde5aa1a36d79d8f8b76116257931"
Jul 06 15:54:23.428503 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:23.428465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp" event={"ID":"5619760d-a735-4d20-9c8e-c83e5ea38129","Type":"ContainerStarted","Data":"229680c48a804d6c419ea9d58cb63d7e93afc3b9cbcc6760c2125277ea49091f"}
Jul 06 15:54:23.428903 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:23.428671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:54:26.554686 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:26.554654    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:26.555077 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:26.554700    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:26.567296 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:26.567264    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:27.462266 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:27.462224    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:54:34.436122 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:34.436088    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-8mrzp"
Jul 06 15:54:38.569317 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:38.569282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:54:53.890029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:54:53.889983    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:55:10.878605 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:55:10.878566    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:55:27.918039 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:55:27.918005    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:55:40.083143 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:55:40.083102    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:55:57.185260 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:55:57.185176    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:56:12.879882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:12.879840    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:56:27.148550 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:27.148512    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:56:28.156064 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:28.156029    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:56:30.163052 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:30.163015    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:56:42.168239 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.168181    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" podUID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" containerName="main" containerID="cri-o://d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c" gracePeriod=45
Jul 06 15:56:42.424322 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.424264    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:56:42.434899 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.434872    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435018 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.434906    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435018 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.434942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435018 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.434976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435186 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435036    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gm7mm\" (UniqueName: \"kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435186 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435073    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm\") pod \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\" (UID: \"e39487e3-8144-42c8-8c21-2ffa91e7b5bb\") "
Jul 06 15:56:42.435186 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435121    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:42.435186 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435131    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home" (OuterVolumeSpecName: "home") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:42.435362 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435217    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache" (OuterVolumeSpecName: "model-cache") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:42.435362 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435356    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:42.435459 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435367    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:42.435459 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.435376    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:42.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.437190    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm" (OuterVolumeSpecName: "kube-api-access-gm7mm") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "kube-api-access-gm7mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:56:42.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.437214    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm" (OuterVolumeSpecName: "dshm") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:42.437231 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.437209    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e39487e3-8144-42c8-8c21-2ffa91e7b5bb" (UID: "e39487e3-8144-42c8-8c21-2ffa91e7b5bb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:56:42.536071 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.536039    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:42.536071 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.536068    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm7mm\" (UniqueName: \"kubernetes.io/projected/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-kube-api-access-gm7mm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:42.536287 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:42.536083    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e39487e3-8144-42c8-8c21-2ffa91e7b5bb-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.158940 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.158901    2566 generic.go:356] "Generic (PLEG): container finished" podID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" containerID="d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c" exitCode=0
Jul 06 15:56:43.159195 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.158978    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"
Jul 06 15:56:43.159195 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.158986    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" event={"ID":"e39487e3-8144-42c8-8c21-2ffa91e7b5bb","Type":"ContainerDied","Data":"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"}
Jul 06 15:56:43.159195 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.159047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m" event={"ID":"e39487e3-8144-42c8-8c21-2ffa91e7b5bb","Type":"ContainerDied","Data":"d1e2e22058e2e81cc5e4f5fec5413810299bc8640415a0fe26fb0c54885b2354"}
Jul 06 15:56:43.159195 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.159071    2566 scope.go:117] "RemoveContainer" containerID="d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"
Jul 06 15:56:43.168427 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.168379    2566 scope.go:117] "RemoveContainer" containerID="d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"
Jul 06 15:56:43.168716 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:56:43.168695    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c\": container with ID starting with d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c not found: ID does not exist" containerID="d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"
Jul 06 15:56:43.168764 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.168725    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c"} err="failed to get container status \"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c\": rpc error: code = NotFound desc = could not find container \"d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c\": container with ID starting with d2338b87749e999c12900a8e8aa9ea99ded7ac86baf7b2b231c018e6aca7557c not found: ID does not exist"
Jul 06 15:56:43.175593 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.175549    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" podUID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" containerName="main" containerID="cri-o://240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9" gracePeriod=45
Jul 06 15:56:43.183695 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.183673    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:56:43.192331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.192313    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-379916/auth-test-model-379916-kserve-65bb7d96ff-dwn2m"]
Jul 06 15:56:43.433055 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.433031    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:56:43.444637 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444612    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444649    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444698    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bm4vd\" (UniqueName: \"kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444733 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444752    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs\") pod \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\" (UID: \"4e1b8798-c2bc-414f-8148-4dc1f9c54b05\") "
Jul 06 15:56:43.444923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444882    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:43.444923 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.444895    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home" (OuterVolumeSpecName: "home") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:43.445095 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.445011    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache" (OuterVolumeSpecName: "model-cache") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:43.445161 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.445142    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.445217 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.445190    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.445217 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.445206    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.446927 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.446894    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm" (OuterVolumeSpecName: "dshm") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:56:43.446927 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.446905    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd" (OuterVolumeSpecName: "kube-api-access-bm4vd") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "kube-api-access-bm4vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:56:43.447059 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.446933    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4e1b8798-c2bc-414f-8148-4dc1f9c54b05" (UID: "4e1b8798-c2bc-414f-8148-4dc1f9c54b05"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:56:43.546030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.545996    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.546030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.546024    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm4vd\" (UniqueName: \"kubernetes.io/projected/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-kube-api-access-bm4vd\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.546030 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.546035    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4e1b8798-c2bc-414f-8148-4dc1f9c54b05-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:56:43.722637 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:43.722555    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" path="/var/lib/kubelet/pods/e39487e3-8144-42c8-8c21-2ffa91e7b5bb/volumes"
Jul 06 15:56:44.166102 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.166065    2566 generic.go:356] "Generic (PLEG): container finished" podID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" containerID="240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9" exitCode=0
Jul 06 15:56:44.166286 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.166112    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"
Jul 06 15:56:44.166286 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.166152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" event={"ID":"4e1b8798-c2bc-414f-8148-4dc1f9c54b05","Type":"ContainerDied","Data":"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"}
Jul 06 15:56:44.166286 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.166196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc" event={"ID":"4e1b8798-c2bc-414f-8148-4dc1f9c54b05","Type":"ContainerDied","Data":"c7cb729da989461dce36d4e063ff90f1a944052dfc57b7d94707dcbee2305a60"}
Jul 06 15:56:44.166286 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.166216    2566 scope.go:117] "RemoveContainer" containerID="240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"
Jul 06 15:56:44.174711 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.174684    2566 scope.go:117] "RemoveContainer" containerID="240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"
Jul 06 15:56:44.175018 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:56:44.174991    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9\": container with ID starting with 240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9 not found: ID does not exist" containerID="240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"
Jul 06 15:56:44.175064 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.175013    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9"} err="failed to get container status \"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9\": rpc error: code = NotFound desc = could not find container \"240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9\": container with ID starting with 240f85df6b147d3071f380d7e5133d6d70110002098a992005e8138ab8e8c8f9 not found: ID does not exist"
Jul 06 15:56:44.192349 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.192318    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:56:44.194234 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:44.194209    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/auth-test-model-650de1-kserve-65c84d44f5-mqwcc"]
Jul 06 15:56:45.657538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:45.657505    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:56:45.721525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:56:45.721487    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" path="/var/lib/kubelet/pods/4e1b8798-c2bc-414f-8148-4dc1f9c54b05/volumes"
Jul 06 15:57:05.884203 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884168    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:57:05.884699 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884681    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" containerName="main"
Jul 06 15:57:05.884791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884702    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" containerName="main"
Jul 06 15:57:05.884791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884713    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" containerName="main"
Jul 06 15:57:05.884791 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884721    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" containerName="main"
Jul 06 15:57:05.884943 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884838    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e1b8798-c2bc-414f-8148-4dc1f9c54b05" containerName="main"
Jul 06 15:57:05.884943 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.884852    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e39487e3-8144-42c8-8c21-2ffa91e7b5bb" containerName="main"
Jul 06 15:57:05.889340 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.889319    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.893746 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.893718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"kube-root-ca.crt\""
Jul 06 15:57:05.893869 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.893739    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"sub-test-model-650de1-kserve-self-signed-certs\""
Jul 06 15:57:05.893869 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.893718    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"openshift-service-ca.crt\""
Jul 06 15:57:05.895200 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.895181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-650de1\"/\"default-dockercfg-zpztf\""
Jul 06 15:57:05.899572 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.899552    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:57:05.931806 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.931771    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.932021 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.931821    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s98qn\" (UniqueName: \"kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.932299 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.932274    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.932636 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.932616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.932810 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.932796    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:05.932929 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:05.932916    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034288 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034288 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034558 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034322    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034558 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034346    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s98qn\" (UniqueName: \"kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034558 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034374    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034558 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.034896 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034865    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.035017 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034910    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.035017 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.034924    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.036737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.036711    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.036968 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.036949    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.044463 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.044442    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s98qn\" (UniqueName: \"kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn\") pod \"sub-test-model-650de1-kserve-5c69b98469-gllpg\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") " pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.199324 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.199226    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:06.358574 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:06.358546    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:57:06.360126 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:57:06.360099    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbec50484_b5e6_4250_8d33_cc58a3443e9f.slice/crio-2484ea6d8572f72a3cf709fbe983aa1552340117f3eddcb8fbdb548cbf15a5fe WatchSource:0}: Error finding container 2484ea6d8572f72a3cf709fbe983aa1552340117f3eddcb8fbdb548cbf15a5fe: Status 404 returned error can't find the container with id 2484ea6d8572f72a3cf709fbe983aa1552340117f3eddcb8fbdb548cbf15a5fe
Jul 06 15:57:07.286146 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:07.286107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" event={"ID":"bec50484-b5e6-4250-8d33-cc58a3443e9f","Type":"ContainerStarted","Data":"7a2fda580009d17fc88829ee7f396cd44ce20ebd32bd8adddf1cc08d360420be"}
Jul 06 15:57:07.286645 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:07.286163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" event={"ID":"bec50484-b5e6-4250-8d33-cc58a3443e9f","Type":"ContainerStarted","Data":"2484ea6d8572f72a3cf709fbe983aa1552340117f3eddcb8fbdb548cbf15a5fe"}
Jul 06 15:57:07.312727 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:07.312683    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" podStartSLOduration=2.312667654 podStartE2EDuration="2.312667654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:57:07.307953845 +0000 UTC m=+2408.088382663" watchObservedRunningTime="2026-07-06 15:57:07.312667654 +0000 UTC m=+2408.093096454"
Jul 06 15:57:09.867635 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:09.867597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:57:16.200228 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:16.200188    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:16.200730 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:16.200265    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:16.213040 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:16.213016    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:16.342444 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:16.342417    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:57:34.487118 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:34.487029    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:57:50.369197 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:50.369163    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:57:53.773636 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:57:53.773599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:58:00.414105 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.414065    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:58:00.417573 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.417553    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.426327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.426306    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"default-dockercfg-bvmvm\""
Jul 06 15:58:00.427430 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.427387    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"openshift-service-ca.crt\""
Jul 06 15:58:00.427538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.427523    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"kube-root-ca.crt\""
Jul 06 15:58:00.428361 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.428342    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-379916\"/\"rate-test-model-379916-kserve-self-signed-certs\""
Jul 06 15:58:00.445833 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.445805    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:58:00.516340 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516307    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.516539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516375    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.516539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.516539 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.516655 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.516655 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.516593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dvbr\" (UniqueName: \"kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617124 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617124 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617130    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617385 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617157    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617385 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617277    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617385 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617316    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dvbr\" (UniqueName: \"kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617555    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617668 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617626    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.617713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.617699    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.619476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.619451    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.619883 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.619862    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.626655 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.626629    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dvbr\" (UniqueName: \"kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr\") pod \"rate-test-model-379916-kserve-764c99b6f9-wwsb4\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") " pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.727317 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.727235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:00.874005 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:00.873980    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:58:00.876310 ip-10-0-135-27 kubenswrapper[2566]: W0706 15:58:00.876275    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0419cb69_ab60_43b1_92ab_20197c864ec3.slice/crio-7a0773fee0d0069372d1f579d37dcfc054a46cd39cc55a304edbe4832457648a WatchSource:0}: Error finding container 7a0773fee0d0069372d1f579d37dcfc054a46cd39cc55a304edbe4832457648a: Status 404 returned error can't find the container with id 7a0773fee0d0069372d1f579d37dcfc054a46cd39cc55a304edbe4832457648a
Jul 06 15:58:01.565737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:01.565700    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" event={"ID":"0419cb69-ab60-43b1-92ab-20197c864ec3","Type":"ContainerStarted","Data":"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"}
Jul 06 15:58:01.565737 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:01.565742    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" event={"ID":"0419cb69-ab60-43b1-92ab-20197c864ec3","Type":"ContainerStarted","Data":"7a0773fee0d0069372d1f579d37dcfc054a46cd39cc55a304edbe4832457648a"}
Jul 06 15:58:01.588476 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:01.588428    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" podStartSLOduration=1.588413852 podStartE2EDuration="1.588413852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 15:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 15:58:01.58721336 +0000 UTC m=+2462.367642180" watchObservedRunningTime="2026-07-06 15:58:01.588413852 +0000 UTC m=+2462.368842631"
Jul 06 15:58:05.393318 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.393270    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" podUID="bec50484-b5e6-4250-8d33-cc58a3443e9f" containerName="main" containerID="cri-o://7a2fda580009d17fc88829ee7f396cd44ce20ebd32bd8adddf1cc08d360420be" gracePeriod=45
Jul 06 15:58:05.589932 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.589896    2566 generic.go:356] "Generic (PLEG): container finished" podID="bec50484-b5e6-4250-8d33-cc58a3443e9f" containerID="7a2fda580009d17fc88829ee7f396cd44ce20ebd32bd8adddf1cc08d360420be" exitCode=0
Jul 06 15:58:05.590111 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.589956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" event={"ID":"bec50484-b5e6-4250-8d33-cc58a3443e9f","Type":"ContainerDied","Data":"7a2fda580009d17fc88829ee7f396cd44ce20ebd32bd8adddf1cc08d360420be"}
Jul 06 15:58:05.648374 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.648311    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:58:05.766424 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766368    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766424 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766430    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s98qn\" (UniqueName: \"kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766489    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766515    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766697 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766864 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766745    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir\") pod \"bec50484-b5e6-4250-8d33-cc58a3443e9f\" (UID: \"bec50484-b5e6-4250-8d33-cc58a3443e9f\") "
Jul 06 15:58:05.766864 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766758    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home" (OuterVolumeSpecName: "home") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:58:05.766864 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766768    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:58:05.767029 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.766989    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:58:05.767159 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.767139    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:05.767278 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.767163    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:05.767278 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.767176    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:05.768703 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.768674    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:58:05.768797 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.768722    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm" (OuterVolumeSpecName: "dshm") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:58:05.768797 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.768750    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn" (OuterVolumeSpecName: "kube-api-access-s98qn") pod "bec50484-b5e6-4250-8d33-cc58a3443e9f" (UID: "bec50484-b5e6-4250-8d33-cc58a3443e9f"). InnerVolumeSpecName "kube-api-access-s98qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:58:05.868448 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.868386    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s98qn\" (UniqueName: \"kubernetes.io/projected/bec50484-b5e6-4250-8d33-cc58a3443e9f-kube-api-access-s98qn\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:05.868448 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.868444    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bec50484-b5e6-4250-8d33-cc58a3443e9f-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:05.868448 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:05.868455    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bec50484-b5e6-4250-8d33-cc58a3443e9f-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:58:06.597534 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:06.597500    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"
Jul 06 15:58:06.597938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:06.597555    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg" event={"ID":"bec50484-b5e6-4250-8d33-cc58a3443e9f","Type":"ContainerDied","Data":"2484ea6d8572f72a3cf709fbe983aa1552340117f3eddcb8fbdb548cbf15a5fe"}
Jul 06 15:58:06.597938 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:06.597605    2566 scope.go:117] "RemoveContainer" containerID="7a2fda580009d17fc88829ee7f396cd44ce20ebd32bd8adddf1cc08d360420be"
Jul 06 15:58:06.628577 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:06.628548    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:58:06.635981 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:06.635960    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-650de1/sub-test-model-650de1-kserve-5c69b98469-gllpg"]
Jul 06 15:58:07.721713 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:07.721679    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bec50484-b5e6-4250-8d33-cc58a3443e9f" path="/var/lib/kubelet/pods/bec50484-b5e6-4250-8d33-cc58a3443e9f/volumes"
Jul 06 15:58:10.727874 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:10.727842    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:10.728355 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:10.727895    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:10.740079 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:10.740058    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:11.634472 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:11.634442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:58:12.963790 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:12.963749    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:58:38.170312 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:38.170275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:58:52.299824 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:58:52.299742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:59:13.175407 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:13.175355    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:59:16.990740 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:16.990703    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:59:28.161831 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:28.161789    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:59:32.010258 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.010201    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" podUID="0419cb69-ab60-43b1-92ab-20197c864ec3" containerName="main" containerID="cri-o://4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85" gracePeriod=45
Jul 06 15:59:32.264862 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.264806    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:59:32.335164 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335126    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335245    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335278    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335303    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5dvbr\" (UniqueName: \"kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335365    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs\") pod \"0419cb69-ab60-43b1-92ab-20197c864ec3\" (UID: \"0419cb69-ab60-43b1-92ab-20197c864ec3\") "
Jul 06 15:59:32.335592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335460    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:32.335592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335532    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home" (OuterVolumeSpecName: "home") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:32.335592 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335542    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache" (OuterVolumeSpecName: "model-cache") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:32.335769 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335746    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-tmp-dir\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:32.335769 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335764    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-home\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:32.335846 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.335778    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-model-cache\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:32.337382 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.337358    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm" (OuterVolumeSpecName: "dshm") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:32.337499 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.337426    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr" (OuterVolumeSpecName: "kube-api-access-5dvbr") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "kube-api-access-5dvbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:59:32.337828 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.337812    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0419cb69-ab60-43b1-92ab-20197c864ec3" (UID: "0419cb69-ab60-43b1-92ab-20197c864ec3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 06 15:59:32.436600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.436567    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0419cb69-ab60-43b1-92ab-20197c864ec3-dshm\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:32.436600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.436595    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dvbr\" (UniqueName: \"kubernetes.io/projected/0419cb69-ab60-43b1-92ab-20197c864ec3-kube-api-access-5dvbr\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:32.436600 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:32.436607    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0419cb69-ab60-43b1-92ab-20197c864ec3-tls-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:33.047070 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.047034    2566 generic.go:356] "Generic (PLEG): container finished" podID="0419cb69-ab60-43b1-92ab-20197c864ec3" containerID="4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85" exitCode=0
Jul 06 15:59:33.047542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.047075    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" event={"ID":"0419cb69-ab60-43b1-92ab-20197c864ec3","Type":"ContainerDied","Data":"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"}
Jul 06 15:59:33.047542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.047098    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"
Jul 06 15:59:33.047542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.047119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4" event={"ID":"0419cb69-ab60-43b1-92ab-20197c864ec3","Type":"ContainerDied","Data":"7a0773fee0d0069372d1f579d37dcfc054a46cd39cc55a304edbe4832457648a"}
Jul 06 15:59:33.047542 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.047140    2566 scope.go:117] "RemoveContainer" containerID="4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"
Jul 06 15:59:33.057085 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.057065    2566 scope.go:117] "RemoveContainer" containerID="4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"
Jul 06 15:59:33.057441 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:59:33.057423    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85\": container with ID starting with 4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85 not found: ID does not exist" containerID="4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"
Jul 06 15:59:33.057523 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.057448    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85"} err="failed to get container status \"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85\": rpc error: code = NotFound desc = could not find container \"4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85\": container with ID starting with 4b064bf53b49d237acc951ca35b9c9168860e108e898b332025d07fee98e8f85 not found: ID does not exist"
Jul 06 15:59:33.070368 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.070341    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:59:33.078209 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.078187    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-379916/rate-test-model-379916-kserve-764c99b6f9-wwsb4"]
Jul 06 15:59:33.721645 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:33.721614    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0419cb69-ab60-43b1-92ab-20197c864ec3" path="/var/lib/kubelet/pods/0419cb69-ab60-43b1-92ab-20197c864ec3/volumes"
Jul 06 15:59:43.027919 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:43.027885    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:59:43.028370 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:43.028120    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy" containerID="cri-o://a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9" gracePeriod=30
Jul 06 15:59:47.087649 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:47.087605    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-tl7gw"]
Jul 06 15:59:48.282885 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.282862    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:59:48.382752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382720    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.382944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382781    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.382944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382809    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.382944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382849    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.382944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.382944 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382921    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-485dc\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.383238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.382990    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.383238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.383238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383097    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert\") pod \"c521b56c-482b-46e3-8cee-f4ac5b235c42\" (UID: \"c521b56c-482b-46e3-8cee-f4ac5b235c42\") "
Jul 06 15:59:48.383238 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383199    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:48.383467 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383327    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:48.383525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383486    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:48.383525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383502    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:59:48.383525 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383499    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-credential-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.383682 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.383533    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.384327 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.384296    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data" (OuterVolumeSpecName: "istio-data") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:48.385433 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.385409    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 15:59:48.385720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.385692    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:48.385816 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.385758    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc" (OuterVolumeSpecName: "kube-api-access-485dc") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "kube-api-access-485dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:59:48.385816 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.385793    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token" (OuterVolumeSpecName: "istio-token") pod "c521b56c-482b-46e3-8cee-f4ac5b235c42" (UID: "c521b56c-482b-46e3-8cee-f4ac5b235c42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:59:48.484290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484259    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-envoy\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484286    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-podinfo\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484295    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c521b56c-482b-46e3-8cee-f4ac5b235c42-istiod-ca-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484302    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484290 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484310    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-workload-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484611 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484317    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c521b56c-482b-46e3-8cee-f4ac5b235c42-istio-data\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:48.484611 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:48.484326    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-485dc\" (UniqueName: \"kubernetes.io/projected/c521b56c-482b-46e3-8cee-f4ac5b235c42-kube-api-access-485dc\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:49.129437 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.129382    2566 generic.go:356] "Generic (PLEG): container finished" podID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerID="a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9" exitCode=0
Jul 06 15:59:49.129623 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.129442    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"
Jul 06 15:59:49.129623 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.129446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerDied","Data":"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"}
Jul 06 15:59:49.129623 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.129490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86" event={"ID":"c521b56c-482b-46e3-8cee-f4ac5b235c42","Type":"ContainerDied","Data":"015a4c46fd9c9be634b4f547c7e6899fd4b3b386df9e02130b9713f44812dd36"}
Jul 06 15:59:49.129623 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.129511    2566 scope.go:117] "RemoveContainer" containerID="a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"
Jul 06 15:59:49.138917 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.138900    2566 scope.go:117] "RemoveContainer" containerID="63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"
Jul 06 15:59:49.147055 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.146878    2566 scope.go:117] "RemoveContainer" containerID="a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"
Jul 06 15:59:49.147131 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:59:49.147112    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9\": container with ID starting with a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9 not found: ID does not exist" containerID="a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"
Jul 06 15:59:49.147173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.147141    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9"} err="failed to get container status \"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9\": rpc error: code = NotFound desc = could not find container \"a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9\": container with ID starting with a49403ee12b379ed71b87248be6a6b5350d2837766830b06cb0123a89abc89c9 not found: ID does not exist"
Jul 06 15:59:49.147173 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.147159    2566 scope.go:117] "RemoveContainer" containerID="63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"
Jul 06 15:59:49.147381 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:59:49.147365    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67\": container with ID starting with 63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67 not found: ID does not exist" containerID="63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"
Jul 06 15:59:49.147452 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.147387    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67"} err="failed to get container status \"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67\": rpc error: code = NotFound desc = could not find container \"63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67\": container with ID starting with 63ff9bedbabcc7dbeedcc5f187cdf841641abb85bfc25977b3ff89c406f87e67 not found: ID does not exist"
Jul 06 15:59:49.162128 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.162101    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:59:49.167642 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.167616    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-379916-openshift-default-b9f97969f-hfr86"]
Jul 06 15:59:49.722229 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:49.722188    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" path="/var/lib/kubelet/pods/c521b56c-482b-46e3-8cee-f4ac5b235c42/volumes"
Jul 06 15:59:51.110538 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:51.110505    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:59:51.110951 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:51.110717    2566 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" containerID="cri-o://633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b" gracePeriod=30
Jul 06 15:59:51.766128 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:51.766070    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f854fbf54-wqcm6_05816ef8-1db3-4de0-a4e2-5c6724f1ae65/authorino/0.log"
Jul 06 15:59:55.891599 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:55.891562    2566 patch_prober.go:28] interesting pod/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 06 15:59:55.891962 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:55.891626    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 06 15:59:56.370466 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.370435    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:59:56.554882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.554792    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.554882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.554829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.554882 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.554861    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.554918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.554980    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555025    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fddh\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555092    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs\") pod \"2dd5c4fd-69be-4f92-9967-ec07284f186d\" (UID: \"2dd5c4fd-69be-4f92-9967-ec07284f186d\") "
Jul 06 15:59:56.555185 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555156    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:56.555541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555237    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:56.555541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555362    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:56.555541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555385    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-credential-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.555541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555425    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 06 15:59:56.555541 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.555437    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-socket\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.556038 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.556009    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data" (OuterVolumeSpecName: "istio-data") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:56.557113 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.557091    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token" (OuterVolumeSpecName: "istio-token") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:59:56.557302 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.557277    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh" (OuterVolumeSpecName: "kube-api-access-2fddh") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "kube-api-access-2fddh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 15:59:56.557365 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.557295    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 06 15:59:56.557365 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.557321    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2dd5c4fd-69be-4f92-9967-ec07284f186d" (UID: "2dd5c4fd-69be-4f92-9967-ec07284f186d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 06 15:59:56.656215 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656186    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dd5c4fd-69be-4f92-9967-ec07284f186d-istiod-ca-cert\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656215 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656211    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-podinfo\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656215 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656220    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fddh\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-kube-api-access-2fddh\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656487 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656230    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-envoy\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656487 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656238    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-workload-certs\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656487 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656245    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-token\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.656487 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.656252    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dd5c4fd-69be-4f92-9967-ec07284f186d-istio-data\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 15:59:56.872992 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:56.872965    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gsq8j_d6f4dab9-f7df-4694-9ae3-8f3a2973c218/manager/0.log"
Jul 06 15:59:57.115254 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.115221    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722530-hcs7s_1969d50c-103b-41f4-90c8-7a5176beff2d/cleanup/0.log"
Jul 06 15:59:57.172342 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.172263    2566 generic.go:356] "Generic (PLEG): container finished" podID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerID="633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b" exitCode=0
Jul 06 15:59:57.172342 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.172300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerDied","Data":"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"}
Jul 06 15:59:57.172342 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.172318    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"
Jul 06 15:59:57.172601 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.172367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr" event={"ID":"2dd5c4fd-69be-4f92-9967-ec07284f186d","Type":"ContainerDied","Data":"f850bf4e218a4d5c47efb893a3173e2d89b820965be7ffa40c3199f3a27f4b0f"}
Jul 06 15:59:57.172601 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.172402    2566 scope.go:117] "RemoveContainer" containerID="633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"
Jul 06 15:59:57.181817 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.181802    2566 scope.go:117] "RemoveContainer" containerID="9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"
Jul 06 15:59:57.190532 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.190513    2566 scope.go:117] "RemoveContainer" containerID="633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"
Jul 06 15:59:57.190803 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:59:57.190781    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b\": container with ID starting with 633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b not found: ID does not exist" containerID="633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"
Jul 06 15:59:57.190880 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.190817    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b"} err="failed to get container status \"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b\": rpc error: code = NotFound desc = could not find container \"633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b\": container with ID starting with 633ce28c3776f823d8362d8ff717b69e35a7a348170211c5c7d545c255258b6b not found: ID does not exist"
Jul 06 15:59:57.190880 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.190853    2566 scope.go:117] "RemoveContainer" containerID="9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"
Jul 06 15:59:57.191122 ip-10-0-135-27 kubenswrapper[2566]: E0706 15:59:57.191104    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193\": container with ID starting with 9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193 not found: ID does not exist" containerID="9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"
Jul 06 15:59:57.191163 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.191129    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193"} err="failed to get container status \"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193\": rpc error: code = NotFound desc = could not find container \"9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193\": container with ID starting with 9ac8ab4df732af56418672ee83d00974aad7a85f7025894f789d605493bdb193 not found: ID does not exist"
Jul 06 15:59:57.197043 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.197021    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:59:57.204519 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.204500    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-650de1-openshift-default-64bfbb547-wqprr"]
Jul 06 15:59:57.227612 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.227588    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722545-2h48w_070be589-012f-4f23-b573-c8c03fbf0628/cleanup/0.log"
Jul 06 15:59:57.613659 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.613632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gt8sw_f0dc9e2e-e399-484e-ad59-d0749c967805/manager/0.log"
Jul 06 15:59:57.722296 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.722263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" path="/var/lib/kubelet/pods/2dd5c4fd-69be-4f92-9967-ec07284f186d/volumes"
Jul 06 15:59:57.855183 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.855152    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-kt6bg_6ced8269-b9bb-47c9-b274-68a24ef89081/copy-manifests/0.log"
Jul 06 15:59:57.868685 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:57.868609    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-kt6bg_6ced8269-b9bb-47c9-b274-68a24ef89081/manager/0.log"
Jul 06 15:59:58.115752 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:58.115723    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-6255s_bc5f7bc1-b314-427d-9768-1d9bffe0fded/postgres/0.log"
Jul 06 15:59:58.928720 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:58.928689    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/util/0.log"
Jul 06 15:59:58.935344 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:58.935325    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/pull/0.log"
Jul 06 15:59:58.942570 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:58.942552    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/extract/0.log"
Jul 06 15:59:59.055331 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.055304    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/util/0.log"
Jul 06 15:59:59.063079 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.063058    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/pull/0.log"
Jul 06 15:59:59.078155 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.078139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/extract/0.log"
Jul 06 15:59:59.205445 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.205342    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/extract/0.log"
Jul 06 15:59:59.213712 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.213691    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/util/0.log"
Jul 06 15:59:59.221974 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.221955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/pull/0.log"
Jul 06 15:59:59.359144 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.359114    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/util/0.log"
Jul 06 15:59:59.373544 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.373518    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/pull/0.log"
Jul 06 15:59:59.391266 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.391249    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/extract/0.log"
Jul 06 15:59:59.510946 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.510853    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f854fbf54-wqcm6_05816ef8-1db3-4de0-a4e2-5c6724f1ae65/authorino/0.log"
Jul 06 15:59:59.646294 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.646258    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8mrzp_5619760d-a735-4d20-9c8e-c83e5ea38129/manager/1.log"
Jul 06 15:59:59.779579 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.779494    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-lmlhw_99c9b368-3f11-4161-b7dc-5335411fe249/manager/0.log"
Jul 06 15:59:59.898948 ip-10-0-135-27 kubenswrapper[2566]: I0706 15:59:59.898922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-2nhqw_5b974909-582b-4e4e-8bfc-dd80e0ed839f/kuadrant-console-plugin/0.log"
Jul 06 16:00:00.023510 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.023484    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gs97p_7c5d7737-644e-4728-b383-8749491d1256/registry-server/0.log"
Jul 06 16:00:00.148824 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.148797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cdl6n_dc46494d-2858-4471-b045-6bc9a40efc20/manager/0.log"
Jul 06 16:00:00.149434 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149414    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-7f9st"]
Jul 06 16:00:00.149730 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149719    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bec50484-b5e6-4250-8d33-cc58a3443e9f" containerName="main"
Jul 06 16:00:00.149770 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149732    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bec50484-b5e6-4250-8d33-cc58a3443e9f" containerName="main"
Jul 06 16:00:00.149770 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149751    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:00.149770 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149757    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:00.149770 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149764    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.149770 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149770    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149777    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149782    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149788    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149793    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149802    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0419cb69-ab60-43b1-92ab-20197c864ec3" containerName="main"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149807    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0419cb69-ab60-43b1-92ab-20197c864ec3" containerName="main"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149857    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0419cb69-ab60-43b1-92ab-20197c864ec3" containerName="main"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149865    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bec50484-b5e6-4250-8d33-cc58a3443e9f" containerName="main"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149870    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149877    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.149934 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.149883    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2dd5c4fd-69be-4f92-9967-ec07284f186d" containerName="istio-proxy"
Jul 06 16:00:00.153883 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.153858    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:00.156799 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.156785    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gjmr6\""
Jul 06 16:00:00.160960 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.160941    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-7f9st"]
Jul 06 16:00:00.182444 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.182401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8hj9\" (UniqueName: \"kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9\") pod \"maas-api-key-cleanup-29722560-7f9st\" (UID: \"6ecdd1bc-79e0-4664-8a4c-915087767ef2\") " pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:00.283590 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.283558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8hj9\" (UniqueName: \"kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9\") pod \"maas-api-key-cleanup-29722560-7f9st\" (UID: \"6ecdd1bc-79e0-4664-8a4c-915087767ef2\") " pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:00.293777 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.293752    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8hj9\" (UniqueName: \"kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9\") pod \"maas-api-key-cleanup-29722560-7f9st\" (UID: \"6ecdd1bc-79e0-4664-8a4c-915087767ef2\") " pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:00.314598 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.314577    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tl7gw_bdc5bd1d-94c5-47a5-9e42-d776340b5c3d/limitador/0.log"
Jul 06 16:00:00.466056 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.465984    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:00.598903 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:00.598882    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722560-7f9st"]
Jul 06 16:00:00.600719 ip-10-0-135-27 kubenswrapper[2566]: W0706 16:00:00.600689    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecdd1bc_79e0_4664_8a4c_915087767ef2.slice/crio-05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048 WatchSource:0}: Error finding container 05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048: Status 404 returned error can't find the container with id 05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048
Jul 06 16:00:01.195659 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:01.195620    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st" event={"ID":"6ecdd1bc-79e0-4664-8a4c-915087767ef2","Type":"ContainerStarted","Data":"4bc87090034002997873e4514f048ecc74a4dc35bbad3492a609a80dcfc14fda"}
Jul 06 16:00:01.195840 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:01.195677    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st" event={"ID":"6ecdd1bc-79e0-4664-8a4c-915087767ef2","Type":"ContainerStarted","Data":"05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048"}
Jul 06 16:00:01.209668 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:01.209648    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577f8bd98b-jm7cf_e54f8ea3-0cfb-49bc-a045-810ef7a004b5/kube-auth-proxy/0.log"
Jul 06 16:00:01.216099 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:01.216054    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st" podStartSLOduration=1.216040982 podStartE2EDuration="1.216040982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:00:01.213750712 +0000 UTC m=+2581.994179509" watchObservedRunningTime="2026-07-06 16:00:01.216040982 +0000 UTC m=+2581.996469761"
Jul 06 16:00:01.583210 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:01.583176    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-k92n7_d14ed339-df5d-4a23-baed-989780818609/payload-processing/0.log"
Jul 06 16:00:02.173819 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:02.173790    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-g4c44_8433c233-fd1f-4cb3-97ea-813a2964d259/main/0.log"
Jul 06 16:00:02.202320 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:02.202283    2566 generic.go:356] "Generic (PLEG): container finished" podID="6ecdd1bc-79e0-4664-8a4c-915087767ef2" containerID="4bc87090034002997873e4514f048ecc74a4dc35bbad3492a609a80dcfc14fda" exitCode=0
Jul 06 16:00:02.202531 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:02.202327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st" event={"ID":"6ecdd1bc-79e0-4664-8a4c-915087767ef2","Type":"ContainerDied","Data":"4bc87090034002997873e4514f048ecc74a4dc35bbad3492a609a80dcfc14fda"}
Jul 06 16:00:02.662772 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:02.662742    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-8mzz5_f8075fa4-6faf-4026-8cb4-39d2b108df52/main/0.log"
Jul 06 16:00:03.337686 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:03.337660    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:03.411555 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:03.411521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s8hj9\" (UniqueName: \"kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9\") pod \"6ecdd1bc-79e0-4664-8a4c-915087767ef2\" (UID: \"6ecdd1bc-79e0-4664-8a4c-915087767ef2\") "
Jul 06 16:00:03.413535 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:03.413514    2566 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9" (OuterVolumeSpecName: "kube-api-access-s8hj9") pod "6ecdd1bc-79e0-4664-8a4c-915087767ef2" (UID: "6ecdd1bc-79e0-4664-8a4c-915087767ef2"). InnerVolumeSpecName "kube-api-access-s8hj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 06 16:00:03.512371 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:03.512281    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8hj9\" (UniqueName: \"kubernetes.io/projected/6ecdd1bc-79e0-4664-8a4c-915087767ef2-kube-api-access-s8hj9\") on node \"ip-10-0-135-27.ec2.internal\" DevicePath \"\""
Jul 06 16:00:04.214221 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:04.214180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st" event={"ID":"6ecdd1bc-79e0-4664-8a4c-915087767ef2","Type":"ContainerDied","Data":"05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048"}
Jul 06 16:00:04.214221 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:04.214219    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722560-7f9st"
Jul 06 16:00:04.214666 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:04.214242    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05c9cb2daf47c045deabf456782136b26617d369269fbd456c5c048e7efea048"
Jul 06 16:00:05.220383 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:05.220348    2566 generic.go:356] "Generic (PLEG): container finished" podID="42729c55-f345-4a32-81e1-5389bec392c7" containerID="383b5b5029fe94c6049d76f62cace32f07d6fd883541373cc4af3149038f2c7e" exitCode=1
Jul 06 16:00:05.220383 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:05.220410    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vk7cf" event={"ID":"42729c55-f345-4a32-81e1-5389bec392c7","Type":"ContainerDied","Data":"383b5b5029fe94c6049d76f62cace32f07d6fd883541373cc4af3149038f2c7e"}
Jul 06 16:00:05.220824 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:05.220707    2566 scope.go:117] "RemoveContainer" containerID="383b5b5029fe94c6049d76f62cace32f07d6fd883541373cc4af3149038f2c7e"
Jul 06 16:00:06.227530 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:06.227499    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-vk7cf" event={"ID":"42729c55-f345-4a32-81e1-5389bec392c7","Type":"ContainerStarted","Data":"89b5586fdc4d7dc77b269a3deb10431e1869127f8482ed58ecc5e115baaf0748"}
Jul 06 16:00:07.114522 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.114486    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjbpc/must-gather-cq762"]
Jul 06 16:00:07.114821 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.114809    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ecdd1bc-79e0-4664-8a4c-915087767ef2" containerName="cleanup"
Jul 06 16:00:07.114871 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.114822    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ecdd1bc-79e0-4664-8a4c-915087767ef2" containerName="cleanup"
Jul 06 16:00:07.114906 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.114881    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c521b56c-482b-46e3-8cee-f4ac5b235c42" containerName="istio-proxy"
Jul 06 16:00:07.114906 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.114892    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ecdd1bc-79e0-4664-8a4c-915087767ef2" containerName="cleanup"
Jul 06 16:00:07.118096 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.118075    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.122034 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.122012    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjbpc\"/\"openshift-service-ca.crt\""
Jul 06 16:00:07.122169 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.122145    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-pjbpc\"/\"default-dockercfg-mpklm\""
Jul 06 16:00:07.123181 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.123163    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-pjbpc\"/\"kube-root-ca.crt\""
Jul 06 16:00:07.129537 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.129513    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/must-gather-cq762"]
Jul 06 16:00:07.140072 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.140047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jhg\" (UniqueName: \"kubernetes.io/projected/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-kube-api-access-c9jhg\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.140162 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.140082    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-must-gather-output\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.240539 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.240508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jhg\" (UniqueName: \"kubernetes.io/projected/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-kube-api-access-c9jhg\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.240906 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.240546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-must-gather-output\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.240906 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.240864    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-must-gather-output\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.252624 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.252594    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jhg\" (UniqueName: \"kubernetes.io/projected/4334dea4-e6a3-42a2-b09f-e79dbbc3e931-kube-api-access-c9jhg\") pod \"must-gather-cq762\" (UID: \"4334dea4-e6a3-42a2-b09f-e79dbbc3e931\") " pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.427452 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.427331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/must-gather-cq762"
Jul 06 16:00:07.566769 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.566743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/must-gather-cq762"]
Jul 06 16:00:07.568165 ip-10-0-135-27 kubenswrapper[2566]: W0706 16:00:07.568126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4334dea4_e6a3_42a2_b09f_e79dbbc3e931.slice/crio-b465d3d6dafcf8d5a13accff2ded97bf89c3300f7cbacf0c0e736b7e212c703c WatchSource:0}: Error finding container b465d3d6dafcf8d5a13accff2ded97bf89c3300f7cbacf0c0e736b7e212c703c: Status 404 returned error can't find the container with id b465d3d6dafcf8d5a13accff2ded97bf89c3300f7cbacf0c0e736b7e212c703c
Jul 06 16:00:07.570197 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:07.570177    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 06 16:00:08.238680 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:08.238644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/must-gather-cq762" event={"ID":"4334dea4-e6a3-42a2-b09f-e79dbbc3e931","Type":"ContainerStarted","Data":"b465d3d6dafcf8d5a13accff2ded97bf89c3300f7cbacf0c0e736b7e212c703c"}
Jul 06 16:00:09.248136 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:09.248027    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/must-gather-cq762" event={"ID":"4334dea4-e6a3-42a2-b09f-e79dbbc3e931","Type":"ContainerStarted","Data":"768f399767d3ec4262fa6a5f29028cc5d6cb562744cff807476754791ef8e859"}
Jul 06 16:00:09.248136 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:09.248096    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/must-gather-cq762" event={"ID":"4334dea4-e6a3-42a2-b09f-e79dbbc3e931","Type":"ContainerStarted","Data":"48fd1e5f8af4d1c9e799049d341ce1991e07ddbc6d8484f9b1a8363f729dd5c5"}
Jul 06 16:00:09.277752 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:09.277681    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjbpc/must-gather-cq762" podStartSLOduration=0.908192289 podStartE2EDuration="2.277658171s" totalImagesPullingTime="1.369465882s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:00:09.274951651 +0000 UTC m=+2590.055380452" watchObservedRunningTime="2026-07-06 16:00:09.277658171 +0000 UTC m=+2590.058086973"
Jul 06 16:00:10.681957 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:10.681922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-92l2l_58a9d157-df15-4431-bd46-f4a4f3e3ea2d/global-pull-secret-syncer/0.log"
Jul 06 16:00:10.930010 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:10.929983    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-74wlp_fb6f9482-c85f-4d8c-952a-d3f575143f09/konnectivity-agent/0.log"
Jul 06 16:00:11.060740 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:11.060662    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-27.ec2.internal_ef6bcc450bef4c891baf1f00f90df783/haproxy/0.log"
Jul 06 16:00:14.967294 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:14.967263    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/extract/0.log"
Jul 06 16:00:15.000221 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.000192    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/util/0.log"
Jul 06 16:00:15.030820 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.030790    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759q98xs_9f50476d-c6c7-4766-bb5a-72a40ebaca96/pull/0.log"
Jul 06 16:00:15.067983 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.067914    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/extract/0.log"
Jul 06 16:00:15.109219 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.109192    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/util/0.log"
Jul 06 16:00:15.143922 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.143895    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e05rm4m_69537b75-910d-4214-bed3-1e61717f1b24/pull/0.log"
Jul 06 16:00:15.174827 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.174801    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/extract/0.log"
Jul 06 16:00:15.199845 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.199815    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/util/0.log"
Jul 06 16:00:15.225387 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.225304    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73ccrzc_53339932-c90e-4667-ad6b-02a8961d03d6/pull/0.log"
Jul 06 16:00:15.258407 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.258366    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/extract/0.log"
Jul 06 16:00:15.290377 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.290327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/util/0.log"
Jul 06 16:00:15.316666 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.316437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef17dv7n_c7143d25-ef83-434a-a4f7-744aeeefcc6b/pull/0.log"
Jul 06 16:00:15.736568 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.736488    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7f854fbf54-wqcm6_05816ef8-1db3-4de0-a4e2-5c6724f1ae65/authorino/0.log"
Jul 06 16:00:15.790032 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.786186    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8mrzp_5619760d-a735-4d20-9c8e-c83e5ea38129/manager/1.log"
Jul 06 16:00:15.790032 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.788133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-8mrzp_5619760d-a735-4d20-9c8e-c83e5ea38129/manager/0.log"
Jul 06 16:00:15.846477 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.846443    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-lmlhw_99c9b368-3f11-4161-b7dc-5335411fe249/manager/0.log"
Jul 06 16:00:15.903870 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.903833    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-2nhqw_5b974909-582b-4e4e-8bfc-dd80e0ed839f/kuadrant-console-plugin/0.log"
Jul 06 16:00:15.975422 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:15.974425    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-gs97p_7c5d7737-644e-4728-b383-8749491d1256/registry-server/0.log"
Jul 06 16:00:16.162262 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:16.162223    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-cdl6n_dc46494d-2858-4471-b045-6bc9a40efc20/manager/0.log"
Jul 06 16:00:16.192653 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:16.192620    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-tl7gw_bdc5bd1d-94c5-47a5-9e42-d776340b5c3d/limitador/0.log"
Jul 06 16:00:18.807183 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:18.807155    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrbfp_23f68635-8369-4b16-9fd7-8582a76f15a2/node-exporter/0.log"
Jul 06 16:00:18.838269 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:18.838194    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrbfp_23f68635-8369-4b16-9fd7-8582a76f15a2/kube-rbac-proxy/0.log"
Jul 06 16:00:18.862099 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:18.862075    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qrbfp_23f68635-8369-4b16-9fd7-8582a76f15a2/init-textfile/0.log"
Jul 06 16:00:20.679145 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.679108    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"]
Jul 06 16:00:20.688871 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.688838    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.692477 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.692447    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"]
Jul 06 16:00:20.778773 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.778733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-proc\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.778961 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.778943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-sys\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.779036 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.779022    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-podres\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.779103 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.779079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7bg\" (UniqueName: \"kubernetes.io/projected/8285832b-da84-43b3-a41a-a59a42ee092a-kube-api-access-fz7bg\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.779199 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.779181    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-lib-modules\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881002 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.880964    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-podres\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881265 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7bg\" (UniqueName: \"kubernetes.io/projected/8285832b-da84-43b3-a41a-a59a42ee092a-kube-api-access-fz7bg\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881382 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881169    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-podres\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881382 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-lib-modules\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881382 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-proc\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881544 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-sys\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881544 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881518    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-sys\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881625 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881538    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-lib-modules\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.881625 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.881569    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8285832b-da84-43b3-a41a-a59a42ee092a-proc\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:20.898782 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:20.898743    2566 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7bg\" (UniqueName: \"kubernetes.io/projected/8285832b-da84-43b3-a41a-a59a42ee092a-kube-api-access-fz7bg\") pod \"perf-node-gather-daemonset-pl6ns\" (UID: \"8285832b-da84-43b3-a41a-a59a42ee092a\") " pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:21.004117 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.004025    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:21.216969 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.215959    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"]
Jul 06 16:00:21.336959 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.336927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns" event={"ID":"8285832b-da84-43b3-a41a-a59a42ee092a","Type":"ContainerStarted","Data":"60693457b28e5b88fe56ce266127d240c01d4e19657d71d111b3c12ee69bf0ae"}
Jul 06 16:00:21.337166 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.336985    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns" event={"ID":"8285832b-da84-43b3-a41a-a59a42ee092a","Type":"ContainerStarted","Data":"efe4f4f1cc6fe72ca1e5f0a06e62d1e5877774a2c7700171a6a5d82df1720454"}
Jul 06 16:00:21.337166 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.337015    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:21.356034 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.355982    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns" podStartSLOduration=1.355968059 podStartE2EDuration="1.355968059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:00:21.355490539 +0000 UTC m=+2602.135919341" watchObservedRunningTime="2026-07-06 16:00:21.355968059 +0000 UTC m=+2602.136396906"
Jul 06 16:00:21.996078 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:21.996054    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cfdb5c74f-rw4lj_c967eb02-f4ec-46df-84cb-e2414569165b/console/0.log"
Jul 06 16:00:23.816129 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:23.816103    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlx8d_84649a02-593c-4194-905b-14bada246537/dns/0.log"
Jul 06 16:00:23.843670 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:23.843645    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-vlx8d_84649a02-593c-4194-905b-14bada246537/kube-rbac-proxy/0.log"
Jul 06 16:00:23.875280 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:23.875253    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-d7dql_fc86a757-c267-46ce-8f42-b3428abd7517/dns-node-resolver/0.log"
Jul 06 16:00:24.613955 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:24.613922    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hhx2_c6eb4b5c-48c6-461d-8350-a42e867d4715/node-ca/0.log"
Jul 06 16:00:25.996621 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:25.996586    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577f8bd98b-jm7cf_e54f8ea3-0cfb-49bc-a045-810ef7a004b5/kube-auth-proxy/0.log"
Jul 06 16:00:26.200203 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:26.200157    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-k92n7_d14ed339-df5d-4a23-baed-989780818609/payload-processing/0.log"
Jul 06 16:00:26.972657 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:26.972632    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jjtrj_0dcfb475-f876-4ebc-927e-ea7e5d4a450c/serve-healthcheck-canary/0.log"
Jul 06 16:00:27.352964 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:27.352938    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-pjbpc/perf-node-gather-daemonset-pl6ns"
Jul 06 16:00:27.788582 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:27.788506    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jv75h_f7127008-cf84-40d0-8d43-e36e77c3608f/kube-rbac-proxy/0.log"
Jul 06 16:00:27.832180 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:27.832150    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jv75h_f7127008-cf84-40d0-8d43-e36e77c3608f/exporter/0.log"
Jul 06 16:00:27.937286 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:27.937262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-jv75h_f7127008-cf84-40d0-8d43-e36e77c3608f/extractor/0.log"
Jul 06 16:00:30.706241 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:30.706127    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gsq8j_d6f4dab9-f7df-4694-9ae3-8f3a2973c218/manager/0.log"
Jul 06 16:00:30.852309 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:30.852281    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722530-hcs7s_1969d50c-103b-41f4-90c8-7a5176beff2d/cleanup/0.log"
Jul 06 16:00:30.893710 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:30.893683    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722545-2h48w_070be589-012f-4f23-b573-c8c03fbf0628/cleanup/0.log"
Jul 06 16:00:30.934730 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:30.934704    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722560-7f9st_6ecdd1bc-79e0-4664-8a4c-915087767ef2/cleanup/0.log"
Jul 06 16:00:31.201166 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.201133    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-gt8sw_f0dc9e2e-e399-484e-ad59-d0749c967805/manager/0.log"
Jul 06 16:00:31.419680 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.419646    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577f8bd98b-jm7cf_e54f8ea3-0cfb-49bc-a045-810ef7a004b5/kube-auth-proxy/0.log"
Jul 06 16:00:31.489866 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.489822    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-kt6bg_6ced8269-b9bb-47c9-b274-68a24ef89081/manager/0.log"
Jul 06 16:00:31.511211 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.511182    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9957545c6-kt6bg_6ced8269-b9bb-47c9-b274-68a24ef89081/copy-manifests/0.log"
Jul 06 16:00:31.544954 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.544917    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-k92n7_d14ed339-df5d-4a23-baed-989780818609/payload-processing/0.log"
Jul 06 16:00:31.603005 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:31.602976    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-6255s_bc5f7bc1-b314-427d-9768-1d9bffe0fded/postgres/0.log"
Jul 06 16:00:33.070640 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:33.070576    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-577f8bd98b-jm7cf_e54f8ea3-0cfb-49bc-a045-810ef7a004b5/kube-auth-proxy/0.log"
Jul 06 16:00:33.186582 ip-10-0-135-27 kubenswrapper[2566]: I0706 16:00:33.186546    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-k92n7_d14ed339-df5d-4a23-baed-989780818609/payload-processing/0.log"